00010000:00010000:1.0:1787827175.478980:0:5567:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.203.101@tcp lock=0x61d2feb329d0cc88: rc = 0 req@ffff8b2e88020780 x1874672524384256/t0(0) o104->8e7ce795-6dad-45e6-b76e-c5bf3324e96e@192.168.203.101@tcp:21/0 lens 328/224 e 0 to 0 dl 1787827186 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.479007:0:5567:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e88020780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524384256:12345-192.168.203.101@tcp:104: Request processed in 74us (209us total) trans 0 rc -22/-22 00000100:00100000:1.0:1787827175.479027:0:5567:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1076 00000100:00100000:1.0:1787827175.479048:0:5567:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524384384 00000100:00100000:1.0:1787827175.479054:0:5567:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524384384 00000100:00100000:1.0:1787827175.479082:0:5567:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1077 00000100:00100000:1.0:1787827175.479093:0:5567:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e88020b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524384384:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.479107:0:5567:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0cc7a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d229fdea expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.479149:0:5567:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.479159:0:5567:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e88020b40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524384384:12345-192.168.203.101@tcp:104: Request processed in 67us (195us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.479171:0:5567:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1077 00010000:00010000:1.0:1787827175.479365:0:14479:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae000/0x61d2feb329d09e6e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d2291ed1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.479404:0:14479:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0cc7a lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d229fdea expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.479416:0:14479:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.479435:0:14479:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0cc7a lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d229fdea expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.479448:0:14479:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0cc7a lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d229fdea expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.479468:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.479472:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.479477:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.479910:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8802a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517629824:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.483819:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.489801:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.494886:0:15045:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787827175.494902:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524384768 00000100:00100000:1.0:1787827175.494909:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524384768 00000100:00100000:1.0:1787827175.494958:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1080 00000100:00100000:1.0:1787827175.494972:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87fca450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524384768:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.494988:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0ccab lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d229feed expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.495042:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.495058:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87fca450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524384768:12345-192.168.203.101@tcp:104: Request processed in 88us (229us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.495084:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1080 00000100:00100000:1.0:1787827175.499807:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524384896 00000100:00100000:1.0:1787827175.499814:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524384896 00000100:00100000:1.0:1787827175.499853:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1081 00000100:00100000:1.0:1787827175.499865:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e88021680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524384896:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.499876:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0ccb9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d229ff3a expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1787827175.499947:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.499955:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e88021680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524384896:12345-192.168.203.101@tcp:104: Request processed in 92us (192us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.499965:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1081 00000100:00100000:1.0:1787827175.505271:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524385024 00000100:00100000:1.0:1787827175.505278:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524385024 00000100:00100000:1.0:1787827175.505325:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1082 00000100:00100000:1.0:1787827175.505337:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e880203c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5839:x1874672524385024:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.505351:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0ccc7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d229ff87 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1787827175.505406:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.505415:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e880203c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5839:x1874672524385024:12345-192.168.203.101@tcp:104: Request processed in 82us (198us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.505427:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1082 00000100:00100000:1.0:1787827175.509798:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88020000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517633920:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.509956:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524385152 00000100:00100000:1.0:1787827175.509963:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524385152 00000100:00100000:1.0:1787827175.510009:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1083 00000100:00100000:1.0:1787827175.510023:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e88020f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5841:x1874672524385152:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.510035:0:5568:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x61d2feb329d0cce3 - lock disappeared 00010000:00010000:1.0:1787827175.510067:0:5568:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.203.101@tcp lock=0x61d2feb329d0cce3: rc = 0 req@ffff8b2e88020f00 x1874672524385152/t0(0) o104->8e7ce795-6dad-45e6-b76e-c5bf3324e96e@192.168.203.101@tcp:21/0 lens 328/224 e 0 to 0 dl 1787827186 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.510084:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e88020f00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5841:x1874672524385152:12345-192.168.203.101@tcp:104: Request processed in 63us (169us total) trans 0 rc -22/-22 00000100:00100000:1.0:1787827175.510098:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1083 00000100:00100000:1.0:1787827175.510582:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88020000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517633920:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.515167:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.520134:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.524623:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.529941:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.529977:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.529998:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524385920 00000100:00100000:1.0:1787827175.530004:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524385920 00000100:00100000:1.0:1787827175.530051:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1088 00000100:00100000:1.0:1787827175.530069:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8802a580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:8207:x1874672524385920:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.530088:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0cd14 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0108 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.530152:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.530166:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8802a580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:8207:x1874672524385920:12345-192.168.203.101@tcp:104: Request processed in 100us (185us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.530195:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1088 00000100:00100000:1.0:1787827175.530474:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880203c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517636864:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.531427:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880203c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517636864:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.535006:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.539678:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.539825:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.544947:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.549422:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.549564:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.554435:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.554972:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88023840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517640576:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.556020:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88023840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517640576:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.559405:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.559881:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88021a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517641344:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.563921:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.564426:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880203c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517641984:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.565308:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880203c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517641984:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.568466:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88020000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517642880:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.568606:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.568740:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.568773:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524387328 00000100:00100000:1.0:1787827175.568779:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524387328 00000100:00100000:1.0:1787827175.568801:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1100 00000100:00100000:1.0:1787827175.568811:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87ff70c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5840:x1874672524387328:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.568820:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0cda0 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a03a8 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787827175.568870:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.568876:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87ff70c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5840:x1874672524387328:12345-192.168.203.101@tcp:104: Request processed in 66us (141us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.568885:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1100 00000100:00100000:1.0:1787827175.569179:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88022d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517643008:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.570062:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88022d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517643008:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.573721:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.578942:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.583348:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.583491:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.588443:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.588482:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.588553:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1105 00000100:00100000:1.0:1787827175.588568:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87ff7c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5839:x1874672524387968:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.588581:0:15045:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x61d2feb329d0cdf4 - lock disappeared 00010000:00010000:1.0:1787827175.588619:0:15045:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.203.101@tcp lock=0x61d2feb329d0cdf4: rc = 0 req@ffff8b2e87ff7c00 x1874672524387968/t0(0) o104->8e7ce795-6dad-45e6-b76e-c5bf3324e96e@192.168.203.101@tcp:21/0 lens 328/224 e 0 to 0 dl 1787827186 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.588640:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87ff7c00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5839:x1874672524387968:12345-192.168.203.101@tcp:104: Request processed in 77us (198us total) trans 0 rc -22/-22 00000100:00100000:1.0:1787827175.588666:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1105 00000100:00100000:1.0:1787827175.588902:0:2347:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1787827175.588915:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880203c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517646208:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.589669:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880203c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517646208:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.595292:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.600182:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.604605:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524388480 00000100:00100000:1.0:1787827175.604613:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524388480 00000100:00100000:1.0:1787827175.604679:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524388608 00000100:00100000:1.0:1787827175.604685:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524388608 00000100:00100000:1.0:1787827175.604711:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1109 00000100:00100000:1.0:1787827175.604727:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87eaf840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5840:x1874672524388480:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.604741:0:15045:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x61d2feb329d0ce25 - lock disappeared 00010000:00010000:1.0:1787827175.604791:0:15045:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.203.101@tcp lock=0x61d2feb329d0ce25: rc = 0 req@ffff8b2e87eaf840 x1874672524388480/t0(0) o104->8e7ce795-6dad-45e6-b76e-c5bf3324e96e@192.168.203.101@tcp:21/0 lens 328/224 e 0 to 0 dl 1787827186 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.604809:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87eaf840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5840:x1874672524388480:12345-192.168.203.101@tcp:104: Request processed in 87us (265us total) trans 0 rc -22/-22 00000100:00100000:1.0:1787827175.604822:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1109 00010000:00010000:1.0:1787827175.604938:0:14479:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827175.604945:0:14479:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0ce17 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x22 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a05df expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.604963:0:14479:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8b2e869af800 already unused, calling callback (ffffffffc19b7990) 00010000:00010000:1.0:1787827175.604967:0:14479:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0ce17 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a05df expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.604984:0:14479:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0ce17 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a05df expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.604996:0:14479:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0ce17 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xb0788cb3d22a05df expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.605013:0:14479:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae000/0x61d2feb329d09e6e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d2291ed1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.605041:0:14479:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0ce17 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a05df expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.605052:0:14479:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.605070:0:14479:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0ce17 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a05df expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.605083:0:14479:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0ce17 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a05df expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.605094:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.605098:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.605100:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.605992:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517648256:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.609336:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.613641:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.613816:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.614249:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8802a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517649920:192.168.203.101@tcp:103:ldlm_bl.0 00010000:00010000:1.0:1787827175.618019:0:14479:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827175.618026:0:14479:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0ce5d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xb0788cb3d22a071a expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.618038:0:14479:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8b2e869af600 already unused, calling callback (ffffffffc19b7990) 00010000:00010000:1.0:1787827175.618044:0:14479:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0ce5d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xb0788cb3d22a071a expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.618074:0:14479:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0ce5d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xb0788cb3d22a071a expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.618085:0:14479:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0ce5d lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xb0788cb3d22a071a expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.618102:0:14479:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0ce4f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a06e2 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827175.618136:0:14479:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0ce5d lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xb0788cb3d22a071a expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.618148:0:14479:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.618174:0:14479:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0ce5d lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xb0788cb3d22a071a expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.618189:0:14479:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0ce5d lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xb0788cb3d22a071a expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.618206:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.618209:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.618213:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.618236:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524389120 00000100:00100000:1.0:1787827175.618243:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524389120 00000100:00100000:1.0:1787827175.618294:0:5568:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787827175.618308:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524389248 00000100:00100000:1.0:1787827175.618312:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524389248 00000100:00100000:1.0:1787827175.618338:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1115 00000100:00100000:1.0:1787827175.618352:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8800d450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:6710:x1874672524389248:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.618367:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0ce4f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a06e2 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.618412:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.618424:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8800d450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:6710:x1874672524389248:12345-192.168.203.101@tcp:104: Request processed in 73us (148us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.618441:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1115 00000100:00100000:1.0:1787827175.619449:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff7840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517650560:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.623053:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.627737:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00010000:00010000:1.0:1787827175.627983:0:14479:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827175.627992:0:14479:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0ce72 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0798 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.628010:0:14479:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8b2e869aec00 already unused, calling callback (ffffffffc19b7990) 00010000:00010000:1.0:1787827175.628016:0:14479:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0ce72 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a0798 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.628025:0:14479:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0ce72 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a0798 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.628039:0:14479:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0ce72 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xb0788cb3d22a0798 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.628063:0:14479:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae000/0x61d2feb329d09e6e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d2291ed1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.628106:0:14479:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0ce72 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a0798 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.628115:0:14479:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.628136:0:14479:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0ce72 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a0798 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.628147:0:14479:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0ce72 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a0798 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.628174:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.628178:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.628183:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.632642:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524389632 00000100:00100000:1.0:1787827175.632661:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524389632 00000100:00100000:1.0:1787827175.632704:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1118 00000100:00100000:1.0:1787827175.632715:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8802a580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5840:x1874672524389632:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.632734:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0ce80 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a07e5 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.632797:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.632805:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8802a580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5840:x1874672524389632:12345-192.168.203.101@tcp:104: Request processed in 92us (216us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.632821:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1118 00010000:00010000:1.0:1787827175.636780:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.636851:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88baba80 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.637893:0:5567:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524389760 00000100:00100000:1.0:1787827175.637901:0:5567:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524389760 00000100:00100000:1.0:1787827175.637960:0:5567:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1119 00000100:00100000:1.0:1787827175.637973:0:5567:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87eca940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524389760:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.637986:0:5567:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0ce8e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0832 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 1 00010000:00010000:1.0:1787827175.638051:0:5567:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.638061:0:5567:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87eca940 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524389760:12345-192.168.203.101@tcp:104: Request processed in 89us (212us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.638079:0:5567:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1119 00000100:00100000:1.0:1787827175.640000:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8802e1c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517653120:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.640067:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.640078:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.640090:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.640115:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.640123:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.640129:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.640132:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.640137:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.640140:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.640148:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.640160:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.640165:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.640168:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.640176:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.640179:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.640198:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.640222:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.640229:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.640237:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.640250:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff61c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517653504:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.640281:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7a80 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.641566:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff61c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517653504:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.641586:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.641599:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.641611:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.641632:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.641686:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.641712:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.641731:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.641754:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.641769:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87ff61c0 x1874672517653504/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.641785:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.641799:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.641812:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.641816:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.641837:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.641845:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.641850:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0cea3 00000080:00010000:1.0:1787827175.641855:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.641858:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0cea3 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.641861:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.641879:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.641918:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.641926:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.641945:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.641956:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.641967:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.641976:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.641979:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.641983:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.641987:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.642001:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.642029:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.642035:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.642047:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.642058:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1063, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.642063:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.642069:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.642074:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.642080:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1063 00010000:00010000:1.0:1787827175.642094:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.642116:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.642139:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.642151:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.642156:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cea3 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a089b expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.642175:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.642184:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff61c0 x1874672517653760/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.642217:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff61c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517653760:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.642264:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7a80 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.643110:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524389888 00000100:00100000:1.0:1787827175.643119:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524389888 00000100:00100000:1.0:1787827175.643181:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1120 00000100:00100000:1.0:1787827175.643195:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87f6bc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524389888:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.643207:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0ce9c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a087f expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1787827175.643271:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.643279:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87f6bc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524389888:12345-192.168.203.101@tcp:104: Request processed in 88us (216us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.643291:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1120 00000100:00100000:1.0:1787827175.645311:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff61c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517653760:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.645361:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.645373:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.645385:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.645396:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.645403:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.645407:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.645410:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.645415:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.645418:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.645426:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.645437:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.645442:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.645445:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.645453:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.645456:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.645464:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.645487:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.645496:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.645503:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.645515:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff6d00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517654144:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.645550:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7a80 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.646733:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff6d00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517654144:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.646768:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.646784:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.646813:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.646826:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.646838:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.646866:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.646876:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.646889:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.646924:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87ff6d00 x1874672517654144/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.646958:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.646977:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.647018:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.647026:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.647053:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.647067:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.647081:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0ceb1 00000080:00010000:1.0:1787827175.647096:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.647104:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0ceb1 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.647111:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.647121:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.647162:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.647175:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.647185:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.647200:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.647220:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.647231:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.647236:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.647246:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.647253:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.647273:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.647288:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.647295:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.647301:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.647317:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1064, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.647324:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.647331:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.647340:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.647348:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1064 00010000:00010000:1.0:1787827175.647369:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.647394:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.647428:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.647448:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.647454:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0ceb1 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a08e8 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.647483:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.647493:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff6d00 x1874672517654400/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.647525:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff6d00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517654400:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.647577:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7a80 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.648630:0:5567:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524390016 00000100:00100000:1.0:1787827175.648637:0:5567:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524390016 00000100:00100000:1.0:1787827175.648731:0:5567:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1121 00000100:00100000:1.0:1787827175.648757:0:5567:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524390016:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.648768:0:5567:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0ceaa lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a08cc expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1787827175.648826:0:5567:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.648837:0:5567:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804f480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524390016:12345-192.168.203.101@tcp:104: Request processed in 94us (254us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.648849:0:5567:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1121 00000100:00100000:1.0:1787827175.650616:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff6d00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517654400:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.650690:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.650702:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.650716:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.650726:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.650734:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.650738:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.650741:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.650759:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.650763:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.650771:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.650782:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.650787:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.650791:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.650803:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.650807:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.650816:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.650843:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.650851:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.650859:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.650873:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff4780 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517654784:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.650930:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7a80 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.651989:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff4780 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517654784:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.652008:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.652032:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.652054:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.652063:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.652072:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.652081:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.652088:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.652099:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.652123:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87ff4780 x1874672517654784/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.652153:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.652185:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.652206:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.652213:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.652255:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.652277:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.652283:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0cebf 00000080:00010000:1.0:1787827175.652293:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.652297:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0cebf from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.652301:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.652314:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.652336:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.652343:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.652350:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.652356:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.652366:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.652369:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.652371:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.652374:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.652377:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.652399:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.652423:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.652431:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.652436:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.652448:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1065, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.652454:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.652462:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.652465:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.652471:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1065 00010000:00010000:1.0:1787827175.652484:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.652502:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.652532:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.652537:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.652539:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d600/0x61d2feb329d0cebf lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0927 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.652556:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.652567:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff4780 x1874672517655040/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.652607:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff4780 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517655040:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.652685:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7a80 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.653690:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524390144 00000100:00100000:1.0:1787827175.653696:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524390144 00000100:00100000:1.0:1787827175.653735:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1122 00000100:00100000:1.0:1787827175.653755:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87fca850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524390144:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.653767:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0ceb8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a090b expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.653811:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.653817:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87fca850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524390144:12345-192.168.203.101@tcp:104: Request processed in 74us (188us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.653823:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1122 00010000:00010000:1.0:1787827175.653920:0:5577:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae000/0x61d2feb329d09e6e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d2291ed1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.653952:0:5577:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0ceb8 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a090b expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.653959:0:5577:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.653978:0:5577:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0ceb8 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a090b expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.653988:0:5577:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0ceb8 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a090b expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.653998:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.653999:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.654002:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.654939:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff7840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517655296:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.658512:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.663812:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.676563:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.681763:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.681956:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00010000:00010000:1.0:1787827175.681977:0:5577:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827175.681986:0:5577:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0cefe lrc: 4/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xb0788cb3d22a0a69 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.681999:0:5577:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8b2e869afe00 already unused, calling callback (ffffffffc19b7990) 00010000:00010000:1.0:1787827175.682006:0:5577:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0cefe lrc: 4/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x409400000000 nid: local remote: 0xb0788cb3d22a0a69 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.682030:0:5577:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0cefe lrc: 4/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x409400000000 nid: local remote: 0xb0788cb3d22a0a69 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.682049:0:5577:0:(ldlm_request.c:1774:ldlm_cli_cancel()) ### lock is already being destroyed 00010000:00010000:1.0:1787827175.682057:0:5577:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0cefe lrc: 3/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c09400000000 nid: local remote: 0xb0788cb3d22a0a69 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.682095:0:5577:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827175.682102:0:5577:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0cef0 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0a31 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.682118:0:5577:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8b2e869afa00 already unused, calling callback (ffffffffc19b7990) 00010000:00010000:1.0:1787827175.682122:0:5577:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0cef0 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a0a31 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.682137:0:5577:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0cef0 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a0a31 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.682148:0:5577:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0cef0 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xb0788cb3d22a0a31 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.682168:0:5577:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae000/0x61d2feb329d09e6e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d2291ed1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.682200:0:5577:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0cef0 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a0a31 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.682208:0:5577:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.682229:0:5577:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0cef0 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a0a31 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.682246:0:5577:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0cef0 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a0a31 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.682262:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.682265:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.682268:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.682285:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517657984:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.683539:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff43c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517657984:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.687052:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.687181:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.687888:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517658880:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.688766:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8804d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517658880:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.692237:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.697415:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.706712:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524391424 00000100:00100000:1.0:1787827175.706721:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524391424 00000100:00100000:1.0:1787827175.706801:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1132 00000100:00100000:1.0:1787827175.706815:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804c780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5840:x1874672524391424:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.706831:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0cf36 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0b8f expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787827175.706930:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.706948:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804c780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5840:x1874672524391424:12345-192.168.203.101@tcp:104: Request processed in 128us (746us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.706962:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1132 00000100:00100000:1.0:1787827175.710255:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8800ad00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517661056:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.710268:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.710279:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.710284:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.710289:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.710294:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.710300:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.710304:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.710309:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.710315:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8800ad00 x1874672517661056/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.710322:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.710329:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.710335:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.710338:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.710358:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.710365:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.710370:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0cf4b 00000080:00010000:1.0:1787827175.710377:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.710381:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0cf4b from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.710385:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.710393:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.710414:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.710421:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.710427:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.710433:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.710439:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.710441:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.710443:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.710446:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.710448:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.710453:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.710458:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.710461:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.710463:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.710470:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1074, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.710472:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.710476:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.710479:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.710482:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1074 00010000:00010000:1.0:1787827175.710492:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.710501:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.710517:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.710522:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.710524:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aee00/0x61d2feb329d0cf4b lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0bea expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.710538:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.710543:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff4b40 x1874672517661440/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.710556:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff4b40 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517661440:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.710597:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7240 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.711757:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524391552 00000100:00100000:1.0:1787827175.711764:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524391552 00000100:00100000:1.0:1787827175.711820:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1133 00000100:00100000:1.0:1787827175.711837:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804cb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524391552:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.711861:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0cf44 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0bce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.711938:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.711947:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804cb40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524391552:12345-192.168.203.101@tcp:104: Request processed in 115us (239us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.711958:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1133 00010000:00010000:1.0:1787827175.711991:0:14480:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 1 total in blp. (1 blwis in pool) 00010000:00010000:1.0:1787827175.712000:0:14480:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0cf44 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0bce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.712010:0:14480:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) Lock ffff8b2e869af600 is referenced, will be cancelled later 00010000:00010000:1.0:1787827175.712015:0:14480:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0cf44 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x22 rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a0bce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.712030:0:14480:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827175.712038:0:14480:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0cf52 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xb0788cb3d22a0c06 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.712061:0:14480:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8b2e869af000 already unused, calling callback (ffffffffc19b7990) 00010000:00010000:1.0:1787827175.712067:0:14480:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0cf52 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xb0788cb3d22a0c06 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.712080:0:14480:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0cf52 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xb0788cb3d22a0c06 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.712089:0:14480:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0cf52 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x408400000000 nid: local remote: 0xb0788cb3d22a0c06 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.712116:0:14480:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae000/0x61d2feb329d09e6e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d2291ed1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.712159:0:14480:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0cf52 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xb0788cb3d22a0c06 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.712179:0:14480:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.712202:0:14480:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0cf52 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xb0788cb3d22a0c06 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.712215:0:14480:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0cf52 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4c09400000020 nid: local remote: 0xb0788cb3d22a0c06 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.712237:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.712241:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.712245:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.712260:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517661824:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.713317:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff5680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517661696:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.713965:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff4b40 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517661440:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.714014:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.714025:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.714036:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.714044:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.714049:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.714053:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.714055:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.714059:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.714061:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.714066:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.714075:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.714078:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.714081:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.714087:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.714090:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.714097:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.714121:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.714128:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.714136:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.714157:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff5680 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517661952:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.714198:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7240 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.715388:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff5680 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517661952:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.715398:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.715410:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.715419:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.715427:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.715435:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.715445:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.715454:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.715461:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.715470:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87ff5680 x1874672517661952/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.715477:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.715485:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.715493:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.715497:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.715521:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.715531:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.715535:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0cf60 00000080:00010000:1.0:1787827175.715541:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.715543:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0cf60 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.715555:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.715566:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.715596:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.715603:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.715610:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.715619:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.715625:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.715629:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.715632:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.715635:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.715637:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.715642:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.715685:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.715690:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.715694:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.715704:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1075, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.715708:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.715712:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.715716:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.715720:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1075 00010000:00010000:1.0:1787827175.715732:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.715757:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.715778:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.715783:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.715785:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0cf60 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0c45 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.715795:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.715799:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff5680 x1874672517662336/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.715808:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff5680 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517662336:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.715842:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7240 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.716853:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524391680 00000100:00100000:1.0:1787827175.716860:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524391680 00000100:00100000:1.0:1787827175.716912:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1134 00000100:00100000:1.0:1787827175.716925:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804d680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524391680:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.716940:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0cf67 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xb0788cb3d22a0c61 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.716991:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.717002:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804d680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5841:x1874672524391680:12345-192.168.203.101@tcp:104: Request processed in 80us (187us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.717025:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1134 00010000:00010000:1.0:1787827175.717142:0:14480:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827175.717157:0:14480:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0cf59 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0c29 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.717172:0:14480:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8b2e869afc00 already unused, calling callback (ffffffffc19b7990) 00010000:00010000:1.0:1787827175.717178:0:14480:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0cf59 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a0c29 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.717200:0:14480:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0cf59 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a0c29 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.717221:0:14480:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0cf59 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xb0788cb3d22a0c29 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.717246:0:14480:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae000/0x61d2feb329d09e6e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d2291ed1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.717291:0:14480:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0cf59 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a0c29 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.717306:0:14480:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.717340:0:14480:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0cf59 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a0c29 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.717354:0:14480:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0cf59 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a0c29 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.717382:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.717386:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.717390:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.721922:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.726417:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.726522:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.731407:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524392320 00000100:00100000:1.0:1787827175.731415:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524392320 00000100:00100000:1.0:1787827175.731460:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1139 00000100:00100000:1.0:1787827175.731477:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804c780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524392320:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.731489:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0cf91 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0d3a expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1787827175.731538:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.731546:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804c780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524392320:12345-192.168.203.101@tcp:104: Request processed in 72us (184us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.731555:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1139 00000100:00100000:1.0:1787827175.736175:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524392448 00000100:00100000:1.0:1787827175.736181:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524392448 00000100:00100000:1.0:1787827175.736230:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1140 00000100:00100000:1.0:1787827175.736246:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804d680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524392448:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.736258:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0cf9f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0d87 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1787827175.736320:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.736331:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804d680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524392448:12345-192.168.203.101@tcp:104: Request processed in 87us (216us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.736343:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1140 00000100:00100000:1.0:1787827175.736584:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517665408:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.737380:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8804de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517665408:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.741028:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88008000 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517665664:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.741053:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.741084:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.741094:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.741111:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.741144:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.741174:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.741188:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.741214:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.741230:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e88008000 x1874672517665664/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.741262:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.741288:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.741305:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.741315:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.741370:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.741383:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.741391:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0cfb4 00000080:00010000:1.0:1787827175.741402:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.741406:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0cfb4 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.741412:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.741422:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.741459:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.741469:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.741478:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.741486:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.741504:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.741507:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.741512:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.741519:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.741523:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.741532:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.741540:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.741545:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.741550:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.741562:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1080, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.741566:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.741571:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.741579:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.741586:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1080 00010000:00010000:1.0:1787827175.741612:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.741639:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.741708:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.741729:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.741735:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0cfb4 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0df0 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.741775:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.741783:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff4000 x1874672517665920/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.741801:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff4000 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517665920:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.741864:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.743341:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524392576 00000100:00100000:1.0:1787827175.743350:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524392576 00000100:00100000:1.0:1787827175.743407:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1141 00000100:00100000:1.0:1787827175.743420:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804c3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5840:x1874672524392576:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.743451:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0cfad lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0dd4 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 13 lru_type: 1 00010000:00010000:1.0:1787827175.743517:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.743526:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804c3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5840:x1874672524392576:12345-192.168.203.101@tcp:104: Request processed in 108us (298us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.743545:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1141 00000100:00100000:1.0:1787827175.746270:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff4000 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517665920:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.746345:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.746361:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.746377:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.746390:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.746397:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.746403:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.746405:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.746410:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.746414:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.746422:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.746434:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.746440:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.746444:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.746453:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.746457:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.746466:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.746501:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.746508:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.746520:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.746535:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff4f00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517666304:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.746590:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.747858:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff4f00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517666304:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.747869:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.747887:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.747899:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.747905:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.747910:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.747916:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.747920:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.747927:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.747934:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87ff4f00 x1874672517666304/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.747944:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.747953:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.747960:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.747970:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.748009:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.748022:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.748028:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0cfc2 00000080:00010000:1.0:1787827175.748038:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.748045:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0cfc2 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.748059:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.748071:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.748100:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.748109:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.748121:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.748139:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.748164:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.748171:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.748173:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.748178:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.748181:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.748189:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.748197:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.748204:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.748209:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.748223:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1081, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.748231:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.748240:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.748248:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.748258:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1081 00010000:00010000:1.0:1787827175.748281:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.748305:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.748342:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.748363:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.748368:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfc2 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0e3d expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.748391:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.748402:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff4f00 x1874672517666560/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.748434:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff4f00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517666560:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.748492:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.749411:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524392704 00000100:00100000:1.0:1787827175.749418:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524392704 00000100:00100000:1.0:1787827175.749462:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1142 00000100:00100000:1.0:1787827175.749475:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804f0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524392704:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.749490:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0cfbb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0e21 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.749535:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.749543:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804f0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524392704:12345-192.168.203.101@tcp:104: Request processed in 71us (176us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.749553:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1142 00000100:00100000:1.0:1787827175.751591:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff4f00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517666560:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.751632:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.751642:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.751671:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.751683:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.751691:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.751696:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.751699:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.751704:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.751708:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.751714:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.751722:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.751725:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.751728:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.751734:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.751736:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.751741:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.751773:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.751778:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.751786:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.751795:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff5a40 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517666944:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.751830:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.752883:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff5a40 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517666944:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.752895:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.752910:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.752917:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.752923:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.752936:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.752943:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.752963:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.752988:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.752998:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87ff5a40 x1874672517666944/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.753027:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.753041:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.753061:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.753070:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.753094:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.753114:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.753121:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0cfd0 00000080:00010000:1.0:1787827175.753132:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.753137:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0cfd0 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.753143:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.753173:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.753207:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.753217:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.753228:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.753254:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.753268:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.753273:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.753279:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.753290:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.753297:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.753325:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.753343:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.753352:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.753359:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.753379:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1082, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.753386:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.753396:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.753402:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.753411:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1082 00010000:00010000:1.0:1787827175.753428:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.753443:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.753475:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.753486:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.753489:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cfd0 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0e8a expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.753510:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.753520:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff5a40 x1874672517667200/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.753545:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff5a40 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517667200:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.753608:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.754676:0:5568:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787827175.754690:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524392832 00000100:00100000:1.0:1787827175.754697:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524392832 00000100:00100000:1.0:1787827175.754752:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1143 00000100:00100000:1.0:1787827175.754767:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804a450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524392832:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.754777:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0cfc9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0e6e expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1787827175.754826:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.754834:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804a450 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524392832:12345-192.168.203.101@tcp:104: Request processed in 70us (227us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.754851:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1143 00000100:00100000:1.0:1787827175.756716:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff5a40 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517667200:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.756777:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.756787:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.756797:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.756808:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.756815:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.756820:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.756823:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.756828:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.756832:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.756840:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.756851:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.756855:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.756859:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.756867:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.756871:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.756880:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.756914:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.756922:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.756930:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.756943:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff43c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517667584:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.756979:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.758300:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff43c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517667584:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.758313:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.758326:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.758337:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.758351:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.758373:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.758384:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.758396:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.758415:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.758430:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87ff43c0 x1874672517667584/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.758449:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.758461:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.758495:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.758502:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.758528:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.758542:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.758546:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0cfde 00000080:00010000:1.0:1787827175.758555:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.758561:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0cfde from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.758567:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.758593:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.758635:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.758672:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.758684:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.758697:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.758715:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.758725:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.758733:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.758741:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.758759:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.758782:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.758806:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.758814:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.758820:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.758834:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1083, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.758841:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.758849:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.758855:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.758864:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1083 00010000:00010000:1.0:1787827175.758891:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.758912:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.758948:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.758967:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.758971:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0cfde lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0ed7 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.758990:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.759008:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff43c0 x1874672517667840/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.759028:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff43c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517667840:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.759077:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.760004:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524392960 00000100:00100000:1.0:1787827175.760010:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524392960 00000100:00100000:1.0:1787827175.760062:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1144 00000100:00100000:1.0:1787827175.760078:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524392960:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.760092:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0cfd7 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0ebb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1787827175.760162:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.760174:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524392960:12345-192.168.203.101@tcp:104: Request processed in 99us (214us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.760187:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1144 00000100:00100000:1.0:1787827175.762441:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff43c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517667840:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.762504:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.762518:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.762532:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.762544:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.762551:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.762556:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.762558:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.762564:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.762568:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.762575:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.762587:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.762592:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.762596:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.762604:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.762608:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.762616:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.762673:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.762681:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.762692:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.762705:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff52c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517668224:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.762766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.764043:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff52c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517668224:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.764062:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.764082:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.764090:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.764104:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.764120:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.764145:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.764159:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.764175:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.764190:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87ff52c0 x1874672517668224/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.764202:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.764214:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.764224:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.764229:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.764258:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.764268:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.764276:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0cfec 00000080:00010000:1.0:1787827175.764285:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.764291:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0cfec from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.764295:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.764310:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.764347:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.764357:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.764366:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.764378:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.764387:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.764394:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.764400:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.764405:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.764411:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.764424:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.764437:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.764442:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.764448:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.764460:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1084, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.764465:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.764470:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.764474:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.764480:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1084 00010000:00010000:1.0:1787827175.764498:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.764533:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.764574:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.764596:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.764601:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0cfec lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0f24 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.764621:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.764631:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff52c0 x1874672517668480/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.764678:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff52c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517668480:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.764729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.765807:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524393088 00000100:00100000:1.0:1787827175.765816:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524393088 00000100:00100000:1.0:1787827175.765862:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1145 00000100:00100000:1.0:1787827175.765875:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804cf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524393088:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.765899:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0cfe5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0f08 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1787827175.765963:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.765971:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804cf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524393088:12345-192.168.203.101@tcp:104: Request processed in 100us (205us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.765982:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1145 00000100:00100000:1.0:1787827175.768018:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff52c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517668480:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.768073:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.768084:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.768096:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.768107:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.768114:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.768119:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.768122:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.768126:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.768130:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.768137:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.768148:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.768152:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.768156:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.768163:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.768165:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.768172:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.768199:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.768206:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.768219:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.768231:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff7c00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517668864:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.768274:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.769417:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff7c00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517668864:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.769427:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.769444:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.769452:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.769459:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.769464:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.769470:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.769479:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.769496:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.769514:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87ff7c00 x1874672517668864/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.769547:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.769567:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.769580:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.769599:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.769627:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.769641:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.769673:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0cffa 00000080:00010000:1.0:1787827175.769685:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.769690:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0cffa from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.769696:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.769709:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.769734:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.769760:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.769771:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.769787:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.769796:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.769803:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.769807:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.769812:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.769816:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.769829:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.769845:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.769850:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.769855:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.769867:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1085, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.769871:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.769876:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.769879:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.769897:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1085 00010000:00010000:1.0:1787827175.769921:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.769943:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.769969:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.769982:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.769987:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0cffa lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0f71 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.770014:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.770025:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff7c00 x1874672517669120/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.770058:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff7c00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517669120:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.770115:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.771016:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524393216 00000100:00100000:1.0:1787827175.771021:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524393216 00000100:00100000:1.0:1787827175.771065:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1146 00000100:00100000:1.0:1787827175.771075:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804da40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5840:x1874672524393216:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.771086:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0cff3 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0f55 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1787827175.771142:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.771149:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804da40 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5840:x1874672524393216:12345-192.168.203.101@tcp:104: Request processed in 75us (172us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.771158:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1146 00010000:00010000:1.0:1787827175.771533:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.773243:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff7c00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517669120:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.773300:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.773310:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.773321:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.773332:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.773338:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.773342:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.773344:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.773349:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.773352:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.773359:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.773369:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.773372:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.773376:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.773384:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.773386:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.773402:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.773427:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.773434:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.773439:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.773449:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff7480 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517669504:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.773482:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.775049:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff7480 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517669504:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.775062:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.775089:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.775113:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.775129:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.775143:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.775166:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.775184:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.775201:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.775213:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87ff7480 x1874672517669504/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.775233:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.775258:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.775285:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.775292:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.775320:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.775335:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.775341:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d008 00000080:00010000:1.0:1787827175.775351:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.775355:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d008 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.775357:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.775369:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.775396:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.775406:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.775419:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.775441:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.775450:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.775454:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.775458:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.775464:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.775468:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.775485:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.775508:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.775515:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.775519:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.775572:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1086, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.775578:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.775582:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.775585:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.775590:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1086 00010000:00010000:1.0:1787827175.775609:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.775625:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.775683:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.775694:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.775697:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d008 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a0fbe expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.775711:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.775718:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff7480 x1874672517669760/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.775734:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff7480 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517669760:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.775794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.776796:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524393344 00000100:00100000:1.0:1787827175.776804:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524393344 00000100:00100000:1.0:1787827175.776852:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1147 00000100:00100000:1.0:1787827175.776862:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804fc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524393344:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.776874:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d001 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0fa2 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.776942:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.776954:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804fc00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524393344:12345-192.168.203.101@tcp:104: Request processed in 92us (216us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.776972:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1147 00000100:00100000:1.0:1787827175.778826:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff7480 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517669760:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.778883:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.778898:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.778924:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.778937:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.778944:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.778949:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.778952:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.778962:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.778966:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.778974:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.778985:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.778989:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.778994:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.779005:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.779009:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.779018:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.779050:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d016 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.779057:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d016 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.779070:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d016 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.779085:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff6940 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517670144:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.779148:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7f00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.781877:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524393472 00000100:00100000:1.0:1787827175.781885:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524393472 00000100:00100000:1.0:1787827175.781928:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1148 00000100:00100000:1.0:1787827175.781940:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804c780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524393472:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.781951:0:15045:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x61d2feb329d0d01d - lock disappeared 00010000:00010000:1.0:1787827175.781994:0:15045:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.203.101@tcp lock=0x61d2feb329d0d01d: rc = 0 req@ffff8b2e8804c780 x1874672524393472/t0(0) o104->8e7ce795-6dad-45e6-b76e-c5bf3324e96e@192.168.203.101@tcp:21/0 lens 328/224 e 0 to 0 dl 1787827186 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.782028:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804c780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524393472:12345-192.168.203.101@tcp:104: Request processed in 88us (190us total) trans 0 rc -22/-22 00000100:00100000:1.0:1787827175.782044:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1148 00000100:00100000:1.0:1787827175.782067:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524393600 00000100:00100000:1.0:1787827175.782072:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524393600 00000100:00100000:1.0:1787827175.782105:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1149 00000100:00100000:1.0:1787827175.782120:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804d680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524393600:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.782135:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d00f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0fe1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.782184:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.782194:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804d680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524393600:12345-192.168.203.101@tcp:104: Request processed in 77us (219us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.782211:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1149 00010000:00010000:1.0:1787827175.782234:0:14479:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827175.782253:0:14479:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d00f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a0fe1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.782268:0:14479:0:(ldlm_lockd.c:1928:ldlm_handle_bl_callback()) Lock ffff8b2e869ae200 is referenced, will be cancelled later 00010000:00010000:1.0:1787827175.782273:0:14479:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d00f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x22 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a0fe1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.782288:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.782292:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.782295:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.782356:0:14480:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827175.782363:0:14480:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d00f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x22 rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a0fe1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.782377:0:14480:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8b2e869ae200 already unused, calling callback (ffffffffc19b7990) 00010000:00010000:1.0:1787827175.782381:0:14480:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d00f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a0fe1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.782394:0:14480:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d00f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a0fe1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.782413:0:14480:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d00f lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xb0788cb3d22a0fe1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.782429:0:14480:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae000/0x61d2feb329d09e6e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d2291ed1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.782454:0:14480:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d00f lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a0fe1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.782461:0:14480:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.782480:0:14480:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d00f lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a0fe1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.782490:0:14480:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d00f lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a0fe1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.782506:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.782509:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.782512:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.782521:0:2347:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00100000:1.0:1787827175.782534:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517670784:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.783843:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff70c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517670784:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.786036:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88008000 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517671040:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.786049:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.786059:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.786065:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.786080:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.786091:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.786107:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.786120:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.786136:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.786150:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e88008000 x1874672517671040/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.786166:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.786191:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.786208:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.786214:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.786240:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.786250:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.786258:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d02b 00000080:00010000:1.0:1787827175.786267:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.786271:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d02b from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.786276:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.786286:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.786319:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.786326:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.786335:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.786350:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.786356:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.786360:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.786362:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.786365:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.786367:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.786372:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.786382:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.786388:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.786392:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.786406:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1088, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.786409:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.786416:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.786420:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.786425:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1088 00010000:00010000:1.0:1787827175.786438:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.786464:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.786500:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.786513:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.786517:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869aea00/0x61d2feb329d0d02b lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a1066 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.786540:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.786555:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff70c0 x1874672517671296/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.786583:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff70c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517671296:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.786633:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7c00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.787634:0:5567:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787827175.787680:0:5567:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524393728 00000100:00100000:1.0:1787827175.787685:0:5567:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524393728 00000100:00100000:1.0:1787827175.787729:0:5567:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1150 00000100:00100000:1.0:1787827175.787739:0:5567:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87fcb850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524393728:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.787761:0:5567:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d024 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a104a expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:1.0:1787827175.787808:0:5567:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.787815:0:5567:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87fcb850 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524393728:12345-192.168.203.101@tcp:104: Request processed in 76us (260us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.787823:0:5567:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1150 00000100:00100000:1.0:1787827175.789673:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff70c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517671296:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.789718:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.789728:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.789738:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.789760:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.789768:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.789773:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.789777:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.789781:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.789786:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.789794:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.789805:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.789810:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.789814:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.789822:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.789825:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.789833:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.789857:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.789862:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.789867:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.789877:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff6940 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517671680:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.789918:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7c00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.791100:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff6940 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517671680:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.791114:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.791135:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.791157:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.791171:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.791182:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.791197:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.791213:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.791228:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.791261:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87ff6940 x1874672517671680/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.791278:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.791297:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.791328:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.791334:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.791358:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.791367:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.791371:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d039 00000080:00010000:1.0:1787827175.791377:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.791382:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d039 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.791388:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.791409:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.791440:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.791448:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.791458:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.791470:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.791483:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.791487:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.791491:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.791495:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.791498:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.791506:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.791519:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.791525:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.791530:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.791546:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1089, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.791552:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.791563:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.791573:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.791581:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1089 00010000:00010000:1.0:1787827175.791607:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.791624:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.791686:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.791699:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.791703:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c000/0x61d2feb329d0d039 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a10b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.791717:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.791723:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff6940 x1874672517671936/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.791755:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff6940 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517671936:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.791807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7c00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.792810:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524393856 00000100:00100000:1.0:1787827175.792816:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524393856 00000100:00100000:1.0:1787827175.792855:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1151 00000100:00100000:1.0:1787827175.792864:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804c3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5839:x1874672524393856:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.792875:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d032 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a1097 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 0 00010000:00010000:1.0:1787827175.792918:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.792927:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804c3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5839:x1874672524393856:12345-192.168.203.101@tcp:104: Request processed in 63us (158us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.792939:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1151 00000100:00100000:1.0:1787827175.794762:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff6940 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517671936:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.794813:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.794825:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.794838:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.794849:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.794855:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.794858:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.794860:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.794865:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.794868:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.794873:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.794881:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.794884:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.794887:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.794893:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.794896:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.794910:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.794928:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.794933:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.794938:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.794946:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff7840 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517672320:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.794979:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7c00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.796022:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87ff7840 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517672320:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.796050:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.796070:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.796080:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.796089:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.796098:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.796118:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.796127:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.796135:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.796145:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87ff7840 x1874672517672320/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.796155:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.796171:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.796178:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.796183:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.796206:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.796216:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.796221:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d047 00000080:00010000:1.0:1787827175.796227:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.796230:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d047 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.796232:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.796240:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.796278:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.796286:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.796293:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.796301:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.796307:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.796311:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.796313:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.796317:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.796319:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.796326:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.796334:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.796338:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.796342:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.796350:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1090, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.796353:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.796358:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.796361:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.796365:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1090 00010000:00010000:1.0:1787827175.796380:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.796393:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.796417:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.796424:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.796426:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c200/0x61d2feb329d0d047 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a10f2 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.796445:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.796451:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87ff7840 x1874672517672704/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.796466:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff7840 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517672704:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.796507:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7c00 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.797560:0:5567:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524393984 00000100:00100000:1.0:1787827175.797568:0:5567:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524393984 00000100:00100000:1.0:1787827175.797674:0:5567:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1153 00000100:00100000:1.0:1787827175.797690:0:5567:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804f0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:8207:x1874672524393984:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.797703:0:5567:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x61d2feb329d0d04e - lock disappeared 00010000:00010000:1.0:1787827175.797741:0:5567:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.203.101@tcp lock=0x61d2feb329d0d04e: rc = 0 req@ffff8b2e8804f0c0 x1874672524393984/t0(0) o104->8e7ce795-6dad-45e6-b76e-c5bf3324e96e@192.168.203.101@tcp:21/0 lens 328/224 e 0 to 0 dl 1787827186 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.797782:0:5567:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804f0c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:8207:x1874672524393984:12345-192.168.203.101@tcp:104: Request processed in 96us (282us total) trans 0 rc -22/-22 00000100:00100000:1.0:1787827175.797797:0:5567:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1153 00010000:00010000:1.0:1787827175.797909:0:14479:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae000/0x61d2feb329d09e6e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d2291ed1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.797947:0:14479:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0d040 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a10d6 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.797955:0:14479:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.797982:0:14479:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0d040 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a10d6 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.797995:0:14479:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af600/0x61d2feb329d0d040 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a10d6 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.798005:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.798007:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.798009:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.802133:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.802177:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.802216:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524394368 00000100:00100000:1.0:1787827175.802223:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524394368 00000100:00100000:1.0:1787827175.802266:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1155 00000100:00100000:1.0:1787827175.802279:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87ff6580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5840:x1874672524394368:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.802309:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d055 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a1131 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827175.802364:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/2 locks to priority blp list, 2 blwis in pool 00000100:00100000:1.0:1787827175.802377:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87ff6580 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5840:x1874672524394368:12345-192.168.203.101@tcp:104: Request processed in 101us (202us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.802390:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1155 00010000:00010000:1.0:1787827175.802419:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.802727:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517673856:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.807445:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00010000:00010000:1.0:1787827175.807806:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.807965:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517674496:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.809183:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8804f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517674496:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.813840:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.814517:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517675136:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.815557:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8804c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517675136:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.819711:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00010000:00010000:1.0:1787827175.824917:0:5578:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827175.824925:0:5578:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d0a2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xb0788cb3d22a12ab expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.824939:0:5578:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8b2e869af000 already unused, calling callback (ffffffffc19b7990) 00010000:00010000:1.0:1787827175.824958:0:5578:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d0a2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xb0788cb3d22a12ab expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.824979:0:5578:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d0a2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xb0788cb3d22a12ab expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.824990:0:5578:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d0a2 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xb0788cb3d22a12ab expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.825012:0:5578:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0d094 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1273 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.825058:0:5578:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d0a2 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xb0788cb3d22a12ab expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.825066:0:5578:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.825094:0:5578:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d0a2 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xb0788cb3d22a12ab expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.825123:0:5578:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d0a2 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xb0788cb3d22a12ab expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.825146:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.825149:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.825153:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.825201:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.825229:0:5567:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524394880 00000100:00100000:1.0:1787827175.825236:0:5567:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524394880 00000100:00100000:1.0:1787827175.825286:0:5567:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524395008 00000100:00100000:1.0:1787827175.825291:0:5567:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524395008 00000100:00100000:1.0:1787827175.825324:0:5567:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1160 00000100:00100000:1.0:1787827175.825338:0:5567:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:8207:x1874672524395008:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.825352:0:5567:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0d094 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a1273 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.825395:0:5567:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.825407:0:5567:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:8207:x1874672524395008:12345-192.168.203.101@tcp:104: Request processed in 69us (138us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.825434:0:5567:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1160 00000100:00100000:1.0:1787827175.825738:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517676672:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.826598:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8804cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517676672:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.830874:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.836738:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.841954:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00010000:00010000:1.0:1787827175.842371:0:5578:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827175.842379:0:5578:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0d0d3 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a13a0 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.842394:0:5578:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8b2e869afc00 already unused, calling callback (ffffffffc19b7990) 00010000:00010000:1.0:1787827175.842402:0:5578:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0d0d3 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a13a0 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.842414:0:5578:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0d0d3 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x420400000000 nid: local remote: 0xb0788cb3d22a13a0 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.842427:0:5578:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0d0d3 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x428400000000 nid: local remote: 0xb0788cb3d22a13a0 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.842454:0:5578:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae000/0x61d2feb329d09e6e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d2291ed1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.842492:0:5578:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0d0d3 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a13a0 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.842500:0:5578:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.842514:0:5578:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0d0d3 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a13a0 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.842523:0:5578:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0d0d3 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a13a0 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.842537:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.842540:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.842543:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.847117:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517679616:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.847271:0:5568:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00000100:00100000:1.0:1787827175.847287:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524395520 00000100:00100000:1.0:1787827175.847294:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524395520 00000100:00100000:1.0:1787827175.847337:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524395648 00000100:00100000:1.0:1787827175.847340:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524395648 00000100:00100000:1.0:1787827175.847376:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1165 00000100:00100000:1.0:1787827175.847390:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804cf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5840:x1874672524395648:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.847407:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d0da lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a13c3 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.847459:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.847472:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804cf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5840:x1874672524395648:12345-192.168.203.101@tcp:104: Request processed in 84us (163us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.847493:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1165 00000100:00100000:1.0:1787827175.847875:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517679744:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.848259:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8804d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517679616:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.848723:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8804c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517679744:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.850859:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88008000 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517680000:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.850875:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.850891:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.850921:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.850933:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.850951:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.850968:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.850982:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.850988:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.850995:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e88008000 x1874672517680000/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.851001:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.851009:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.851029:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.851035:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.851062:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.851076:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.851083:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d0f6 00000080:00010000:1.0:1787827175.851094:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.851101:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d0f6 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.851105:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.851112:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.851138:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.851152:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.851162:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.851170:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.851179:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.851189:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.851195:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.851202:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.851206:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.851216:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.851224:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.851229:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.851233:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.851242:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1100, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.851245:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.851250:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.851254:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.851258:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1100 00010000:00010000:1.0:1787827175.851273:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.851290:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.851317:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.851324:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.851328:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d0f6 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a143a expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.851342:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.851347:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87fd9e00 x1874672517680384/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.851361:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fd9e00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517680384:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.851400:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7840 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.852411:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524395776 00000100:00100000:1.0:1787827175.852417:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524395776 00000100:00100000:1.0:1787827175.852454:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1166 00000100:00100000:1.0:1787827175.852462:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804c3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524395776:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.852472:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d0ef lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a141e expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.852513:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.852519:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804c3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524395776:12345-192.168.203.101@tcp:104: Request processed in 58us (163us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.852524:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1166 00000100:00100000:1.0:1787827175.854570:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fd9e00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517680384:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.854635:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.854669:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.854684:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.854698:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.854706:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.854711:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.854714:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.854720:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.854723:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.854732:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.854759:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.854766:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.854770:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.854781:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.854785:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.854795:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.854828:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.854836:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.854844:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.854858:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fd83c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517680640:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.854905:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7840 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.856457:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fd83c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517680640:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.856470:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.856499:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.856509:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.856523:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.856535:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.856555:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.856569:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.856577:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.856600:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87fd83c0 x1874672517680640/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.856625:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.856722:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.856737:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.856753:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.856776:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.856785:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.856790:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d104 00000080:00010000:1.0:1787827175.856798:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.856802:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d104 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.856806:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.856820:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.856852:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.856863:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.856880:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.856892:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.856901:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.856906:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.856911:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.856919:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.856928:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.856940:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.856953:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.856960:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.856966:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.856981:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1101, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.856989:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.856998:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.857003:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.857010:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1101 00010000:00010000:1.0:1787827175.857026:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.857047:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.857098:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.857110:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.857116:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d104 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a1487 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.857153:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.857164:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87fd83c0 x1874672517680896/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.857198:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fd83c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517680896:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.857249:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7840 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.858194:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.864397:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.870264:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.870863:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517682304:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.876891:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.882551:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.883182:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517683584:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.886468:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87f652c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517683840:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.886477:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.886487:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.886495:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.886503:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.886513:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.886522:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.886529:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.886536:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.886546:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87f652c0 x1874672517683840/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.886555:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.886565:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.886574:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.886576:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.886600:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.886608:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.886612:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d14a 00000080:00010000:1.0:1787827175.886619:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.886622:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d14a from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.886626:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.886634:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.886684:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.886693:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.886700:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.886706:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.886710:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.886713:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.886715:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.886717:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.886719:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.886724:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.886739:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.886753:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.886757:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.886766:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1106, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.886781:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.886786:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.886788:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.886792:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1106 00010000:00010000:1.0:1787827175.886808:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.886818:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.886837:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.886843:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.886846:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af200/0x61d2feb329d0d14a lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a15fa expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.886862:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.886869:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87fdbc00 x1874672517684224/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.886891:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fdbc00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517684224:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.886931:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7000 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.888825:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524396544 00000100:00100000:1.0:1787827175.888834:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524396544 00000100:00100000:1.0:1787827175.888899:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1172 00000100:00100000:1.0:1787827175.888915:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804cf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:8207:x1874672524396544:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.888932:0:5568:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x61d2feb329d0d151 - lock disappeared 00010000:00010000:1.0:1787827175.888972:0:5568:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.203.101@tcp lock=0x61d2feb329d0d151: rc = 0 req@ffff8b2e8804cf00 x1874672524396544/t0(0) o104->8e7ce795-6dad-45e6-b76e-c5bf3324e96e@192.168.203.101@tcp:21/0 lens 328/224 e 0 to 0 dl 1787827186 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.888992:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804cf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:8207:x1874672524396544:12345-192.168.203.101@tcp:104: Request processed in 83us (223us total) trans 0 rc -22/-22 00000100:00100000:1.0:1787827175.889014:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1172 00000100:00100000:1.0:1787827175.893827:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517685248:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.894016:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524396800 00000100:00100000:1.0:1787827175.894023:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524396800 00000100:00100000:1.0:1787827175.894068:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1174 00000100:00100000:1.0:1787827175.894079:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804d680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:6710:x1874672524396800:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.894088:0:5568:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x61d2feb329d0d166 - lock disappeared 00010000:00010000:1.0:1787827175.894138:0:5568:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.203.101@tcp lock=0x61d2feb329d0d166: rc = 0 req@ffff8b2e8804d680 x1874672524396800/t0(0) o104->8e7ce795-6dad-45e6-b76e-c5bf3324e96e@192.168.203.101@tcp:21/0 lens 328/224 e 0 to 0 dl 1787827186 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.894168:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804d680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:6710:x1874672524396800:12345-192.168.203.101@tcp:104: Request processed in 91us (193us total) trans 0 rc -22/-22 00000100:00100000:1.0:1787827175.894199:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1174 00010000:00010000:1.0:1787827175.894406:0:6036:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae000/0x61d2feb329d09e6e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d2291ed1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.894435:0:6036:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0d158 lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a1639 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787827175.894441:0:6036:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.894457:0:6036:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0d158 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a1639 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787827175.894479:0:6036:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afc00/0x61d2feb329d0d158 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a1639 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787827175.894505:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.894509:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.894513:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.899129:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524397056 00000100:00100000:1.0:1787827175.899136:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524397056 00000100:00100000:1.0:1787827175.899185:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1176 00000100:00100000:1.0:1787827175.899201:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5839:x1874672524397056:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.899219:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d17b lrc: 4/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400400000000 nid: local remote: 0xb0788cb3d22a16cc expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.899282:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.899296:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:5839:x1874672524397056:12345-192.168.203.101@tcp:104: Request processed in 97us (205us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.899311:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1176 00000100:00100000:1.0:1787827175.904787:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00010000:00010000:1.0:1787827175.910303:0:6036:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827175.910311:0:6036:0:(ldlm_lockd.c:1907:ldlm_handle_bl_callback()) ### client blocking AST callback handler ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d19e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xb0788cb3d22a1774 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.910324:0:6036:0:(ldlm_lockd.c:1921:ldlm_handle_bl_callback()) Lock ffff8b2e869af200 already unused, calling callback (ffffffffc19b7990) 00010000:00010000:1.0:1787827175.910331:0:6036:0:(ldlm_request.c:1732:ldlm_cli_convert()) ### client lock convert START ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d19e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xb0788cb3d22a1774 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.910359:0:6036:0:(ldlm_request.c:1742:ldlm_cli_convert()) ### client lock convert END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d19e lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x400000000 nid: local remote: 0xb0788cb3d22a1774 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.910374:0:6036:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d19e lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xb0788cb3d22a1774 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.910390:0:6036:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d190 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a173c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.910425:0:6036:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d19e lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xb0788cb3d22a1774 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.910437:0:6036:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.910459:0:6036:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d19e lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 5 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xb0788cb3d22a1774 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.910478:0:6036:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d19e lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xb0788cb3d22a1774 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.910496:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.910500:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.910503:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827175.910677:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a940 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517687680:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.910721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.911693:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524397440 00000100:00100000:1.0:1787827175.911699:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524397440 00000100:00100000:1.0:1787827175.911740:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1179 00000100:00100000:1.0:1787827175.911765:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87eaf840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5839:x1874672524397440:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.911774:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d190 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a173c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.911823:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.911842:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87eaf840 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5839:x1874672524397440:12345-192.168.203.101@tcp:104: Request processed in 79us (205us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.911861:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1179 00000100:00100000:1.0:1787827175.913512:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a940 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517687680:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.913557:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.913565:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.913574:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.913583:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.913591:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.913596:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.913598:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.913603:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.913606:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.913614:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.913630:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.913633:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.913637:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.913687:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.913691:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.913700:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.913720:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.913724:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.913729:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.913739:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fda580 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517688064:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.913795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.914366:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517687808:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.914380:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.914391:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.914398:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.914407:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.914416:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.914425:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.914431:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.914437:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.914447:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517687808/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 576/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.914463:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.914473:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.914481:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.914485:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787827175.914514:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.914517:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1a5 00000080:00010000:2.0:1787827175.914521:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.914524:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1a5 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.914527:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.914534:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.914551:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.914558:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.914568:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.914570:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.914580:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.914590:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.914599:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.914603:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.914610:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.914618:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.914623:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.914626:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.914629:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.914632:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.914634:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.914637:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.914643:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.914692:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.914699:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.914706:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.914714:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.914720:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.914724:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.914726:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.914729:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.914732:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.914739:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.914761:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.914765:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.914769:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.914775:0:21402:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), flags=9 00000080:00200000:2.0:1787827175.914778:0:21402:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), intent=getattr 00000002:00010000:2.0:1787827175.914785:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile0,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.914802:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.914808:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.914815:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.914826:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517688192:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.914860:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2780 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.915011:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fda580 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517688064:192.168.203.101@tcp:101:createmany.0 00010000:00010000:1.0:1787827175.915019:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.915026:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.915030:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.915038:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.915043:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.915049:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.915065:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.915073:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:1.0:1787827175.915082:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87fda580 x1874672517688064/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:1.0:1787827175.915092:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:1.0:1787827175.915100:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.915106:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.915109:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:1.0:1787827175.915126:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:1.0:1787827175.915132:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.915136:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d1ac 00000080:00010000:1.0:1787827175.915143:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.915146:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1ac from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.915149:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.915157:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:1.0:1787827175.915180:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.915187:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.915194:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.915203:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.915209:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.915213:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.915215:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.915219:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.915222:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.915230:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.915242:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.915245:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.915247:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.915254:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1111, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.915257:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.915261:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.915264:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.915269:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1111 00010000:00010000:1.0:1787827175.915282:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.915305:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.915326:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.915333:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.915335:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1ac lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a17b3 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.915348:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.915355:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87fda580 x1874672517688448/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.915368:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fda580 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517688448:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.915404:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.915895:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517688192:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.915905:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.915923:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.915937:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.915944:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.915953:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.915962:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.915969:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.915976:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.915983:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517688192/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.915992:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.916000:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.916012:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.916020:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.916027:0:21402:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.916034:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.916040:0:21402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d1b3 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xb0788cb3d22a17cf expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.916055:0:21402:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1787827175.916059:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.916075:0:21402:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8): rc = 0 00000080:00200000:2.0:1787827175.916082:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.916087:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1a5 00000080:00010000:2.0:1787827175.916096:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.916101:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1a5 from it ffffb4b781ec3b20 00010000:00010000:2.0:1787827175.916107:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.916123:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200008:2.0:1787827175.916169:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.916179:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.916188:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.916196:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827175.916209:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827175.916216:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.916220:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.916222:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1a5 00000080:00010000:2.0:1787827175.916225:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1a5 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.916228:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827175.916235:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787827175.916248:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.916254:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.916263:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.916273:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.916284:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.916308:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.916333:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.916339:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.916352:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.916363:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.916370:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.916374:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.916377:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.916381:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.916385:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.916388:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000100:00100000:1.0:1787827175.916389:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524397568 00000080:00200000:2.0:1787827175.916396:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1787827175.916397:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524397568 00000080:00200000:2.0:1787827175.916422:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.916433:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000100:00100000:1.0:1787827175.916436:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1180 00010000:00010000:2.0:1787827175.916447:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.916451:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87eae1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524397568:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.916461:0:5568:0:(ldlm_lockd.c:2448:ldlm_callback_handler()) callback on lock 0x61d2feb329d0d1b3 - lock disappeared 00010000:00010000:2.0:1787827175.916472:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.916487:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.916492:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.916495:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.916501:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:1.0:1787827175.916504:0:5568:0:(ldlm_lockd.c:2345:ldlm_callback_errmsg()) @@@ Operate with invalid parameter, NID=12345-192.168.203.101@tcp lock=0x61d2feb329d0d1b3: rc = 0 req@ffff8b2e87eae1c0 x1874672524397568/t0(0) o104->8e7ce795-6dad-45e6-b76e-c5bf3324e96e@192.168.203.101@tcp:21/0 lens 328/224 e 0 to 0 dl 1787827186 ref 1 fl Interpret:/0/0 rc -22/-22 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827175.916507:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.916530:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.916533:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87eae1c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:8207:x1874672524397568:12345-192.168.203.101@tcp:104: Request processed in 85us (196us total) trans 0 rc -22/-22 00000080:00200000:2.0:1787827175.916551:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000100:00100000:1.0:1787827175.916552:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1180 00000080:00200000:2.0:1787827175.916560:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.916567:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.916574:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.916581:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.916588:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.916596:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.916602:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.916611:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787827175.916627:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.916636:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.916642:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1a5 00000080:00010000:2.0:1787827175.916673:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1a5 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.916679:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x22 rrc: 3 type: IBT gid 0 flags: 0x10420400000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.916690:0:21402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d1a5 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x22 rrc: 3 type: IBT gid 0 flags: 0x10420400000000 nid: local remote: 0xb0788cb3d22a1797 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.916706:0:21402:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000080:00200000:2.0:1787827175.916712:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.916717:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.916725:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.916730:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.916740:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.916768:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.916784:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.916789:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.916797:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.916806:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.916809:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.916812:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.916814:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.916817:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.916819:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.916822:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.916829:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.916843:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.916851:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.916859:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.916869:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.916890:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.916895:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.916900:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.916904:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.916907:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.916920:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.916933:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.916938:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.916942:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.916954:0:21402:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), flags=9 00000080:00200000:2.0:1787827175.916962:0:21402:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), intent=getattr 00000002:00010000:2.0:1787827175.916970:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile0,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.916995:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.917001:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.917009:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.917021:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517688832:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.917046:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b29c0 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.918551:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fda580 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517688448:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.918598:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.918607:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.918616:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.918623:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.918628:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.918631:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.918633:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.918636:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.918639:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.918669:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.918678:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.918682:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.918685:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.918691:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.918693:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.918699:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.918716:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.918721:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.918730:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.918755:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fd92c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517688960:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.918797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.919587:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517688832:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.919599:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.919611:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.919617:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.919627:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.919636:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.919670:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.919678:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.919685:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.919694:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517688832/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.919703:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.919713:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.919720:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.919724:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.919762:0:21402:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8): rc = 0 00000080:00200000:2.0:1787827175.919772:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.919776:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1ba 00000080:00010000:2.0:1787827175.919782:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.919784:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1ba from it ffffb4b781ec3b20 00010000:00010000:2.0:1787827175.919793:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.919805:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1787827175.919825:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.919831:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.919839:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.919851:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.919859:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.919865:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.919870:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.919872:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1ba 00000080:00010000:2.0:1787827175.919876:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1ba from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.919887:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.919897:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.919910:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.919917:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.919934:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.919937:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.919949:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.919959:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.919970:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.919974:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.919982:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.919991:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.919998:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.920001:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.920004:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.920007:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.920010:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.920013:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.920022:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1787827175.920038:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fd92c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517688960:192.168.203.101@tcp:101:createmany.0 00000080:00200000:2.0:1787827175.920046:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00010000:00010000:1.0:1787827175.920050:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.920054:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.920061:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.920080:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.920095:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.920109:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.920113:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.920115:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00010000:00010000:1.0:1787827175.920120:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1787827175.920121:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.920126:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.920140:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.920141:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.920158:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827175.920166:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00010000:00010000:1.0:1787827175.920171:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827175.920171:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.920177:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.920183:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00010000:00010000:1.0:1787827175.920187:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200008:2.0:1787827175.920190:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.920196:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00010000:00010000:1.0:1787827175.920205:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.920205:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.920216:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000002:00100000:1.0:1787827175.920222:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87fd92c0 x1874672517688960/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1787827175.920228:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000002:00010000:1.0:1787827175.920241:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827175.920242:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.920249:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.920254:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1ba 00000002:00010000:1.0:1787827175.920258:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1787827175.920259:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1ba from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.920264:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827175.920276:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.920285:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1787827175.920288:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827175.920299:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.920304:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000080:00200000:1.0:1787827175.920308:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000008:00010000:2.0:1787827175.920310:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.920313:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.920321:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.920329:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.920330:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.920338:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d1c1 00000020:00200000:2.0:1787827175.920341:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.920345:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000080:00010000:1.0:1787827175.920346:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.920353:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1c1 from it ffffb4b781c77ba0 00010000:00010000:2.0:1787827175.920360:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.920361:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787827175.920379:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1787827175.920382:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.920384:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.920390:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.920396:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.920399:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.920402:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.920404:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.920411:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787827175.920417:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.920426:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000080:00200000:2.0:1787827175.920432:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00010000:00010000:1.0:1787827175.920436:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000002:00010000:2.0:1787827175.920441:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.920446:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.920450:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.920460:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00010000:00010000:2.0:1787827175.920465:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.920471:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.920475:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:2.0:1787827175.920476:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.920483:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:1.0:1787827175.920484:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:2.0:1787827175.920489:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00000080:00010000:1.0:1787827175.920492:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:2.0:1787827175.920494:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.920498:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.920506:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.920512:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.920518:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.920525:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.920528:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.920531:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200000:1.0:1787827175.920533:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200008:2.0:1787827175.920535:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:1.0:1787827175.920540:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:2.0:1787827175.920542:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000080:00200000:1.0:1787827175.920547:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000002:00010000:2.0:1787827175.920551:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.920558:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827175.920559:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1112, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.920568:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00010000:00010000:2.0:1787827175.920571:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827175.920574:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.920582:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:2.0:1787827175.920584:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00200000:1.0:1787827175.920590:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1112 00000080:00010000:2.0:1787827175.920590:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.920595:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1ba 00000080:00010000:2.0:1787827175.920601:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1ba from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.920603:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827175.920610:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.920612:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.920625:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.920630:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00010000:00010000:1.0:1787827175.920634:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00000008:00010000:2.0:1787827175.920636:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.920640:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.920667:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.920678:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.920687:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.920691:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.920699:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.920708:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1787827175.920711:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.920712:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.920717:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.920723:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00010000:00010000:1.0:1787827175.920727:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000020:00010000:2.0:1787827175.920728:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00010000:00010000:1.0:1787827175.920732:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8de00/0x61d2feb329d0d1c1 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a181c expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00010000:2.0:1787827175.920733:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.920739:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.920758:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000020:00100000:1.0:1787827175.920770:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000080:00200000:2.0:1787827175.920774:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000100:00100000:1.0:1787827175.920779:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87fd92c0 x1874672517689216/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000002:00010000:2.0:1787827175.920783:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.920794:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.920804:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fd92c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517689216:192.168.203.101@tcp:36:createmany.0 00010000:00010000:2.0:1787827175.920806:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.920815:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.920818:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.920821:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.920824:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.920827:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.920834:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.920842:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.920845:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.920848:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.920851:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.920855:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000100:00100000:1.0:1787827175.920861:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000080:00200000:2.0:1787827175.920861:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.920871:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.920877:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.920885:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787827175.920891:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.920894:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.920896:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1ba 00000080:00010000:2.0:1787827175.920899:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1ba from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.920901:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.920908:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.920921:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.920925:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.920931:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.920933:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.920939:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.920947:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.920955:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.920958:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.920965:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.920973:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.920977:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.920980:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.920982:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.920985:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.920988:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.920991:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.920996:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.921012:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.921019:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.921026:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921034:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.921039:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.921043:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.921045:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.921048:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.921051:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921057:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.921064:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.921068:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.921070:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.921073:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.921076:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.921080:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.921085:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.921090:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.921101:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.921106:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.921109:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.921111:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1ba 00000080:00010000:2.0:1787827175.921114:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1ba from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.921116:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.921122:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787827175.921131:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.921134:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.921139:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.921140:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.921146:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921153:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.921160:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.921163:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921174:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.921181:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.921184:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.921186:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.921188:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.921190:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.921192:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.921194:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.921198:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.921209:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.921214:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.921219:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921226:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.921232:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.921235:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.921237:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.921240:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.921242:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921252:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.921259:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.921262:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.921264:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.921266:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.921270:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.921273:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.921278:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.921283:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787827175.921290:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787827175.921295:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.921298:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.921300:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1ba 00000080:00010000:2.0:1787827175.921303:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1ba from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.921305:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787827175.921311:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1787827175.921323:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.921326:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.921331:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.921332:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.921338:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921345:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.921352:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.921354:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921360:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.921367:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.921370:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.921372:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.921374:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.921376:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.921378:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.921380:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.921384:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.921396:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.921401:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.921405:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921412:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.921418:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.921421:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.921423:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.921426:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.921428:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921434:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.921441:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.921444:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.921446:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.921448:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.921452:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.921456:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.921460:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.921465:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1787827175.921475:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1787827175.921481:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.921483:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.921485:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1ba 00000080:00010000:2.0:1787827175.921488:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1ba from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.921490:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1787827175.921496:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1787827175.921504:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.921507:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.921512:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.921513:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.921518:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921526:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.921532:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.921534:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921545:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.921552:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.921555:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.921557:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.921559:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.921561:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.921564:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.921566:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.921570:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.921581:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.921586:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.921591:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921598:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.921604:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.921607:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.921609:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.921613:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.921615:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921626:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.921634:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.921637:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.921639:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.921642:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.921667:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.921673:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.921680:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.921686:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1787827175.921694:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1787827175.921700:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.921704:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.921707:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1ba 00000080:00010000:2.0:1787827175.921710:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1ba from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.921716:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1787827175.921726:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1787827175.921740:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.921757:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000100:00100000:1.0:1787827175.921760:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524397824 00000008:00010000:2.0:1787827175.921764:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000100:00100000:1.0:1787827175.921767:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524397824 00000008:00010000:2.0:1787827175.921767:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.921777:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921786:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.921793:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.921796:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921803:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.921807:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1182 00000100:00100000:1.0:1787827175.921818:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804cf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5839:x1874672524397824:12345-192.168.203.101@tcp:104: 00000020:00200000:2.0:1787827175.921819:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.921825:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.921829:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00010000:00010000:1.0:1787827175.921830:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afa00/0x61d2feb329d0d1ba lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a1800 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00000020:00010000:2.0:1787827175.921834:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.921840:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.921845:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.921847:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.921853:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.921868:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.921874:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.921880:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.921884:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00010000:00010000:2.0:1787827175.921890:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.921891:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804cf00 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5839:x1874672524397824:12345-192.168.203.101@tcp:104: Request processed in 76us (180us total) trans 0 rc 0/0 00000080:00200000:2.0:1787827175.921907:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000100:00100000:1.0:1787827175.921911:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1182 00000080:00010000:2.0:1787827175.921913:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.921917:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.921921:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.921925:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.921932:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.921941:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.921945:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.921949:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.921954:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.921959:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.921969:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.921978:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.921985:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 30 res 8589935617/30 (0 0) 00010000:00010000:2.0:1787827175.922014:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.922027:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.922035:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.922048:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517689344:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.922088:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b29c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.922157:0:2349:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2349:1874672517689472:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787827175.923046:0:2349:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2349:1874672517689472:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.923732:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fd92c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517689216:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.923798:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.923809:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.923820:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.923830:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.923836:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.923840:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.923842:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.923847:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.923850:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.923858:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.923868:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.923872:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.923876:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.923884:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.923888:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.923897:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.923921:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.923929:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.923936:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.923949:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fda1c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517689600:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.923986:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.924778:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517689344:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.924792:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.924804:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.924813:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.924825:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.924835:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.924851:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.924859:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.924867:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.924879:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517689344/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 576/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.924889:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.924901:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.924909:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.924928:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787827175.924965:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.924970:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1c8 00000080:00010000:2.0:1787827175.924973:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.924976:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1c8 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.924979:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.924992:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.925017:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.925027:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.925041:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.925045:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.925058:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.925070:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.925083:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.925087:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.925096:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.925106:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.925114:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000100:00100000:1.0:1787827175.925119:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fda1c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517689600:192.168.203.101@tcp:101:createmany.0 00000020:00010000:2.0:1787827175.925120:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.925126:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.925130:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00010000:00010000:1.0:1787827175.925134:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1787827175.925135:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.925142:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.925154:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1787827175.925161:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.925171:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.925179:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827175.925187:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00010000:00010000:1.0:1787827175.925188:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.925198:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.925200:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.925209:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.925211:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.925232:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.925232:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000002:00100000:1.0:1787827175.925246:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87fda1c0 x1874672517689600/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787827175.925248:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.925257:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000002:00010000:1.0:1787827175.925259:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1787827175.925263:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.925270:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.925275:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000002:00010000:1.0:1787827175.925284:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.925286:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.925307:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.925309:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.925316:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.925317:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.925323:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.925334:0:21402:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), flags=9 00000080:00200000:2.0:1787827175.925337:0:21402:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), intent=getattr 00000080:00200000:1.0:1787827175.925341:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000002:00010000:2.0:1787827175.925347:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile0,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00000080:00200000:1.0:1787827175.925350:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:1.0:1787827175.925355:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d1cf 00000080:00010000:1.0:1787827175.925361:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.925363:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1cf from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.925366:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.925374:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.925379:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.925398:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.925406:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000080:00200000:1.0:1787827175.925410:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.925418:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000100:00100000:2.0:1787827175.925423:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517689728:192.168.203.101@tcp:101:statmany.0 00010000:00010000:1.0:1787827175.925426:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.925433:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.925437:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.925440:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.925442:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.925445:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.925447:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.925452:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000100:00100000:2.0:1787827175.925457:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b29c0 going to sleep for 11 seconds 00000080:00200000:1.0:1787827175.925465:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.925470:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.925473:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.925480:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1113, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:1.0:1787827175.925482:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.925485:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:1.0:1787827175.925487:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.925490:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1113 00010000:00010000:1.0:1787827175.925502:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.925517:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.925535:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.925539:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.925541:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1cf lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a185b expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:1.0:1787827175.925555:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:1.0:1787827175.925560:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87fda1c0 x1874672517689984/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827175.925570:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fda1c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517689984:192.168.203.101@tcp:36:createmany.0 00000100:00100000:1.0:1787827175.925599:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.926483:0:5567:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524397952 00000100:00100000:1.0:1787827175.926490:0:5567:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524397952 00000100:00100000:1.0:1787827175.926536:0:5567:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1183 00000100:00100000:1.0:1787827175.926546:0:5567:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804c3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524397952:12345-192.168.203.101@tcp:104: 00010000:00010000:1.0:1787827175.926554:0:5567:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1c8 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a183f expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.926589:0:5567:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.926593:0:5567:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804c3c0 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524397952:12345-192.168.203.101@tcp:104: Request processed in 50us (150us total) trans 0 rc 0/0 00000100:00100000:1.0:1787827175.926599:0:5567:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1183 00000100:00100000:1.0:1787827175.926853:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517690112:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.927636:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8804d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517690112:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.928214:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fda1c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517689984:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.928266:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.928278:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.928289:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.928300:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.928307:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.928312:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.928315:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.928320:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.928324:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.928332:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.928342:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.928346:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.928350:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.928359:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.928362:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.928368:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.928386:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.928391:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.928397:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.928407:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fdb840 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517690240:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.928440:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.929218:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517689728:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.929232:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.929242:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.929250:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.929259:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.929268:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.929277:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.929289:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.929296:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.929305:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517689728/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.929316:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.929326:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.929335:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.929339:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.929366:0:21402:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8): rc = 0 00000080:00200000:2.0:1787827175.929375:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.929379:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1d6 00000080:00010000:2.0:1787827175.929385:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.929388:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1d6 from it ffffb4b781ec3b20 00010000:00010000:2.0:1787827175.929391:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.929398:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1787827175.929432:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.929440:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.929449:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.929456:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.929465:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.929471:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.929475:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.929477:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1d6 00000080:00010000:2.0:1787827175.929480:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1d6 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.929483:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.929489:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.929502:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.929509:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.929523:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.929525:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.929537:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.929551:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.929560:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.929563:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.929570:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.929578:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.929583:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.929586:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.929589:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.929591:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.929595:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.929597:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.929604:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.929628:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.929635:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.929642:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.929673:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.929684:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.929688:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.929691:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.929694:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.929697:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.929705:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.929713:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.929718:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.929723:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.929733:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.929739:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.929755:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.929762:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.929770:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827175.929777:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000100:00100000:1.0:1787827175.929785:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fdb840 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517690240:192.168.203.101@tcp:101:createmany.0 00000080:00200000:2.0:1787827175.929785:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.929793:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.929797:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1d6 00010000:00010000:1.0:1787827175.929798:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1787827175.929803:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1d6 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.929810:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827175.929811:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.929830:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.929831:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827175.929849:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00010000:00010000:1.0:1787827175.929850:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000001:00010000:2.0:1787827175.929854:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.929863:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00010000:00010000:1.0:1787827175.929864:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000008:00010000:2.0:1787827175.929867:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:1.0:1787827175.929878:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.929887:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.929905:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.929907:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.929923:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787827175.929930:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.929936:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000002:00100000:1.0:1787827175.929947:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87fdb840 x1874672517690240/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1787827175.929950:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000002:00010000:1.0:1787827175.929971:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787827175.929972:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.929979:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.929986:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.929988:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000002:00010000:1.0:1787827175.929990:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1787827175.929992:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.929996:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.929998:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.930005:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1787827175.930008:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.930013:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.930023:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000080:00200000:1.0:1787827175.930031:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000002:00010000:2.0:1787827175.930032:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000080:00200000:1.0:1787827175.930039:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00010000:00010000:2.0:1787827175.930042:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00010000:1.0:1787827175.930044:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d1dd 00000080:00010000:1.0:1787827175.930051:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.930055:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1dd from it ffffb4b781c77ba0 00010000:00010000:2.0:1787827175.930057:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.930058:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827175.930079:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00010000:00010000:1.0:1787827175.930080:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787827175.930086:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.930091:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.930095:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.930099:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930108:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.930117:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.930121:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000080:00200000:2.0:1787827175.930122:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.930128:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.930132:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000002:00010000:1.0:1787827175.930133:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000080:00200008:2.0:1787827175.930139:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00010000:00010000:1.0:1787827175.930143:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.930145:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.930157:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.930162:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930166:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827175.930173:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.930184:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00010000:00010000:2.0:1787827175.930184:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1787827175.930188:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.930197:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00000080:00200000:2.0:1787827175.930198:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.930201:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.930203:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1d6 00000080:00010000:2.0:1787827175.930207:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1d6 from it ffffb4b781ec3cb8 00010000:00010000:1.0:1787827175.930211:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930212:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827175.930231:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.930237:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.930248:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000080:00200000:1.0:1787827175.930251:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000001:00010000:2.0:1787827175.930254:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000080:00200000:1.0:1787827175.930258:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000008:00010000:2.0:1787827175.930263:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000080:00200000:1.0:1787827175.930264:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000008:00010000:2.0:1787827175.930268:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00000080:00200000:1.0:1787827175.930277:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1114, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00010000:00010000:2.0:1787827175.930278:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.930284:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.930291:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00010000:00010000:2.0:1787827175.930295:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.930296:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.930302:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1114 00000020:00200000:2.0:1787827175.930312:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.930316:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.930324:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.930325:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.930347:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1787827175.930348:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.930352:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.930359:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.930365:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.930368:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.930371:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.930373:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.930380:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1787827175.930384:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827175.930392:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.930395:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c800/0x61d2feb329d0d1dd lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a18a8 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.930398:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.930409:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000020:00100000:1.0:1787827175.930414:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00010000:00010000:2.0:1787827175.930418:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.930423:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87fdb840 x1874672517690496/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827175.930434:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.930443:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000100:00100000:1.0:1787827175.930445:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fdb840 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517690496:192.168.203.101@tcp:36:createmany.0 00000080:00010000:2.0:1787827175.930448:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.930452:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.930454:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.930457:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930463:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.930469:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.930472:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.930475:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.930477:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.930480:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.930483:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.930490:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00000100:00100000:1.0:1787827175.930494:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00010000:00010000:2.0:1787827175.930497:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.930507:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787827175.930514:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.930517:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.930520:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1d6 00000080:00010000:2.0:1787827175.930525:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1d6 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.930528:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.930535:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.930549:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.930552:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.930558:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.930559:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.930565:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930571:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.930576:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.930578:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930583:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.930587:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.930590:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.930591:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.930592:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.930594:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.930595:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.930597:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.930601:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.930613:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.930618:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.930622:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930627:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.930630:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.930632:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.930634:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.930636:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.930638:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930642:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.930678:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.930681:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.930685:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.930687:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.930692:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.930696:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.930702:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.930706:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.930714:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.930718:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.930720:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.930722:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1d6 00000080:00010000:2.0:1787827175.930724:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1d6 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.930726:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.930730:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787827175.930735:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.930738:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.930754:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.930756:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.930763:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930769:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.930774:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.930775:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930784:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.930789:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.930791:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.930793:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.930794:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.930795:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.930797:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.930798:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.930803:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.930813:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.930817:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.930820:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930825:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.930829:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.930831:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.930832:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.930835:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.930837:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930849:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.930857:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.930861:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.930864:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.930867:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.930872:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.930876:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.930883:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.930888:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787827175.930893:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787827175.930897:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.930901:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.930903:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1d6 00000080:00010000:2.0:1787827175.930906:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1d6 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.930909:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787827175.930916:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1787827175.930932:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.930937:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.930943:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.930945:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.930952:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930960:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.930968:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.930971:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.930978:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.930986:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.930990:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.930992:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.930994:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.931005:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.931012:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.931014:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.931019:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.931032:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.931038:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.931044:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.931052:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.931058:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.931062:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.931065:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.931068:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.931071:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.931078:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.931086:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.931090:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.931093:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.931096:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.931100:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.931105:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.931111:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.931118:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1787827175.931131:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1787827175.931137:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.931141:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.931143:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1d6 00000080:00010000:2.0:1787827175.931147:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1d6 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.931150:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1787827175.931157:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1787827175.931168:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.931172:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.931179:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.931181:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.931188:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.931197:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.931205:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.931207:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.931220:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.931228:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.931232:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.931234:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.931236:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.931239:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.931242:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.931244:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.931250:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.931265:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.931272:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.931280:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.931288:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.931295:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.931298:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.931301:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.931305:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.931308:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.931320:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.931330:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.931334:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.931338:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.931341:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.931347:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.931352:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.931359:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.931366:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1787827175.931375:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1787827175.931382:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.931386:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.931389:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1d6 00000080:00010000:2.0:1787827175.931394:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1d6 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.931398:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1787827175.931412:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1787827175.931432:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.931436:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000100:00100000:1.0:1787827175.931443:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524398080 00000008:00010000:2.0:1787827175.931444:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.931448:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00000100:00100000:1.0:1787827175.931450:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524398080 00010000:00010000:2.0:1787827175.931457:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.931468:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.931475:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.931478:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.931486:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1184 00010000:00010000:2.0:1787827175.931487:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.931496:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524398080:12345-192.168.203.101@tcp:104: 00000020:00200000:2.0:1787827175.931501:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1787827175.931505:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869afe00/0x61d2feb329d0d1d6 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a188c expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1787827175.931508:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.931514:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.931518:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.931520:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.931523:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.931525:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.931531:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.931548:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00010000:00010000:1.0:1787827175.931551:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.931556:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524398080:12345-192.168.203.101@tcp:104: Request processed in 63us (156us total) trans 0 rc 0/0 00000002:00010000:2.0:1787827175.931556:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000100:00100000:1.0:1787827175.931568:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1184 00010000:00010000:2.0:1787827175.931569:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.931580:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.931590:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.931598:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.931604:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.931609:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.931615:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.931634:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.931674:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.931679:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.931682:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.931684:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.931689:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.931695:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.931702:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.931712:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 30 res 8589935617/30 (0 0) 00010000:00010000:2.0:1787827175.931757:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.931770:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.931790:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.931806:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517690624:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.931854:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b29c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.932825:0:2349:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2349:1874672517690752:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.933379:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fdb840 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517690496:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.933425:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.933433:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.933442:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.933450:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.933455:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.933458:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.933460:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.933464:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.933475:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.933484:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.933493:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.933498:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.933503:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.933512:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.933516:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.933525:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.933550:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.933558:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.933577:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.933587:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fdad00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517690880:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.933622:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.934291:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517690624:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.934303:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934316:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934323:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934333:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934343:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934353:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934361:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934368:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.934378:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517690624/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 576/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.934388:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.934399:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934407:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.934412:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787827175.934442:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.934445:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1e4 00000080:00010000:2.0:1787827175.934447:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.934449:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1e4 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.934451:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934457:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.934473:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.934479:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.934488:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.934490:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.934498:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.934505:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.934512:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.934514:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.934520:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.934528:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.934534:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.934537:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.934540:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.934543:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.934545:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.934548:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.934555:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.934585:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.934593:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.934605:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.934611:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.934615:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.934618:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.934620:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.934622:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.934624:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.934629:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.934634:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.934636:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.934639:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.934644:0:21402:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), flags=9 00000080:00200000:2.0:1787827175.934666:0:21402:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), intent=getattr 00000002:00010000:2.0:1787827175.934672:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile0,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.934687:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934692:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934697:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.934706:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517691008:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.934756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b29c0 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.934955:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fdad00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517690880:192.168.203.101@tcp:101:createmany.0 00010000:00010000:2.0:1787827175.934965:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934974:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934981:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934988:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.934997:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935006:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935013:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935025:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.935042:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87fdad00 x1874672517690880/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.935052:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.935062:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935071:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.935075:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.935093:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:2.0:1787827175.935101:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.935105:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d1eb 00000080:00010000:2.0:1787827175.935111:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:2.0:1787827175.935114:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1eb from it ffffb4b781c77ba0 00010000:00010000:2.0:1787827175.935117:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935125:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.935147:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:2.0:1787827175.935153:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.935160:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.935169:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.935176:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.935179:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:2.0:1787827175.935182:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:2.0:1787827175.935185:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:2.0:1787827175.935194:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.935202:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.935210:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.935214:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:2.0:1787827175.935217:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.935226:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1115, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:2.0:1787827175.935229:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:2.0:1787827175.935233:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:2.0:1787827175.935237:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:2.0:1787827175.935239:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1115 00010000:00010000:2.0:1787827175.935249:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.935258:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.935274:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.935279:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787827175.935281:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8da00/0x61d2feb329d0d1eb lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a18e7 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1787827175.935295:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827175.935306:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e8803e580 x1874672517691264/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827175.935320:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803e580 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517691264:192.168.203.101@tcp:36:createmany.0 00000100:00100000:2.0:1787827175.935354:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2780 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.935857:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517691008:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.935865:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935871:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935876:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935881:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935888:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935894:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935898:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935902:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.935908:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517691008/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.935916:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.935925:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935934:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.935943:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.935954:0:21402:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935961:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935968:0:21402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae400/0x61d2feb329d0d1f2 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xb0788cb3d22a1903 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.935986:0:21402:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1787827175.935994:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.936013:0:21402:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8): rc = 0 00000080:00200000:2.0:1787827175.936024:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.936029:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1e4 00000080:00010000:2.0:1787827175.936037:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.936043:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1e4 from it ffffb4b781ec3b20 00010000:00010000:2.0:1787827175.936053:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.936069:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200008:2.0:1787827175.936088:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.936095:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.936104:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.936112:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827175.936121:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.936128:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.936132:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.936136:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1e4 00000080:00010000:2.0:1787827175.936139:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1e4 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.936143:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827175.936150:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827175.936163:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.936173:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.936185:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.936188:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.936198:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.936209:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.936219:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.936223:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.936238:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.936247:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000020:00200000:2.0:1787827175.936248:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.936256:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.936259:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.936262:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.936265:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.936268:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.936271:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.936278:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.936299:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.936307:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.936315:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.936325:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.936335:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.936341:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.936344:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.936348:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.936352:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.936366:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.936382:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.936387:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.936394:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.936398:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.936405:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.936412:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.936423:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00000100:00100000:1.0:1787827175.936452:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000100:00100000:1.0:1787827175.936474:0:5568:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524398336 00000100:00100000:1.0:1787827175.936477:0:5568:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524398336 00000100:00100000:1.0:1787827175.936493:0:5568:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1186 00000100:00100000:1.0:1787827175.936498:0:5568:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87fd8780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5839:x1874672524398336:12345-192.168.203.101@tcp:104: 00010000:00010000:2.0:1787827175.936619:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 6 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.936633:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787827175.936641:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.936665:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.936671:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1e4 00000080:00010000:2.0:1787827175.936680:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1e4 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.936687:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.936698:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.936719:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.936727:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.936734:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.936735:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.936755:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.936763:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.936769:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.936772:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.936778:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.936784:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.936787:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.936789:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.936790:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.936792:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.936793:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.936795:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.936799:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.936814:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00010000:00010000:1.0:1787827175.936817:0:5568:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000002:00010000:2.0:1787827175.936819:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.936826:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.936841:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.936849:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.936853:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.936856:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.936860:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.936863:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.936871:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.936879:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.936883:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.936887:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.936890:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.936893:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00010000:00010000:1.0:1787827175.936896:0:5568:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000080:00200000:2.0:1787827175.936897:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.936906:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00000100:00100000:1.0:1787827175.936907:0:5568:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87fd8780 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_001:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5839:x1874672524398336:12345-192.168.203.101@tcp:104: Request processed in 406us (454us total) trans 0 rc 0/0 00010000:00010000:2.0:1787827175.936916:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000100:00100000:1.0:1787827175.936924:0:5568:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1186 00010000:00010000:2.0:1787827175.936936:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.936943:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.936948:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.936951:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1e4 00000080:00010000:2.0:1787827175.936956:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1e4 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.936960:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10420000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.936968:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af800/0x61d2feb329d0d1e4 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10420000000000 nid: local remote: 0xb0788cb3d22a18cb expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.936977:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.936982:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.936988:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.936990:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.936998:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.937004:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.937010:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.937012:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.937021:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.937026:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.937031:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.937036:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.937040:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.937044:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.937048:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.937051:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.937058:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.937069:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.937074:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.937079:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.937088:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.937094:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.937098:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.937103:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.937111:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.937116:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.937143:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.937155:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.937160:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.937166:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.937177:0:21402:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), flags=9 00000080:00200000:2.0:1787827175.937181:0:21402:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), intent=getattr 00000002:00010000:2.0:1787827175.937190:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile0,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.937218:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.937229:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.937240:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.937257:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517691648:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.937291:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2b40 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.938825:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803e580 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517691264:192.168.203.101@tcp:36:createmany.0 00000080:00200000:2.0:1787827175.938882:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:2.0:1787827175.938893:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.938904:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.938925:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.938937:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.938943:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:2.0:1787827175.938946:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:2.0:1787827175.938951:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:2.0:1787827175.938955:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.938963:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.938973:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.938977:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:2.0:1787827175.938981:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.938990:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:2.0:1787827175.938993:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:2.0:1787827175.939001:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.939024:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d200 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.939031:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d200 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.939039:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afa00/0x61d2feb329d0d200 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.939053:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803d2c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517691776:192.168.203.101@tcp:101:createmany.0 00000100:00100000:2.0:1787827175.939095:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2780 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.939810:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517691648:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.939819:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.939827:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.939833:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.939840:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.939849:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.939858:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.939866:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.939873:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.939892:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517691648/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.939902:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.939914:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.939923:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.939927:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.939960:0:21402:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8): rc = 0 00000080:00200000:2.0:1787827175.939970:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.939974:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1f9 00000080:00010000:2.0:1787827175.939981:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.939983:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1f9 from it ffffb4b781ec3b20 00010000:00010000:2.0:1787827175.939985:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.939990:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1787827175.940013:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.940018:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.940024:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.940029:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.940034:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.940038:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.940042:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.940043:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1f9 00000080:00010000:2.0:1787827175.940046:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1f9 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.940048:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.940052:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.940062:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.940067:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.940083:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.940085:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.940098:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940107:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.940118:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.940122:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940130:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.940140:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.940146:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.940150:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.940153:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.940156:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.940160:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.940163:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.940171:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.940203:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.940212:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.940223:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940235:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.940245:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.940253:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.940260:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.940266:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.940272:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940291:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.940315:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.940325:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.940328:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.940333:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.940337:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.940341:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.940347:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.940353:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827175.940360:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.940370:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.940377:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.940380:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1f9 00000080:00010000:2.0:1787827175.940386:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1f9 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.940392:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827175.940412:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827175.940422:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.940425:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.940431:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.940432:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.940437:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940445:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.940457:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.940459:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940463:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.940468:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.940473:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.940478:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.940482:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.940485:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.940488:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.940490:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.940497:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.940518:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.940524:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.940530:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940536:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.940554:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.940559:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.940562:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.940568:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.940572:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940580:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.940593:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.940597:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.940606:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.940611:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.940616:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.940623:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.940633:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.940642:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827175.940676:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827175.940681:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.940685:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.940687:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1f9 00000080:00010000:2.0:1787827175.940691:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1f9 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.940696:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827175.940713:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787827175.940721:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.940724:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.940729:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.940730:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.940736:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940754:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.940762:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.940767:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940788:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.940795:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.940799:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.940801:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.940802:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.940804:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.940808:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.940811:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.940819:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.940833:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.940839:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.940846:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940856:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.940864:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.940867:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.940869:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.940871:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.940873:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940878:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.940883:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.940885:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.940888:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.940890:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.940894:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.940899:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.940909:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.940916:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.940926:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787827175.940931:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.940935:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.940938:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1f9 00000080:00010000:2.0:1787827175.940941:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1f9 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.940944:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.940952:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.940961:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.940965:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.940971:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.940973:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.940980:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.940987:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.940994:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.940996:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941007:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.941014:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.941017:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.941020:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.941021:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.941024:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.941026:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.941029:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.941035:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.941048:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.941054:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.941060:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941068:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.941074:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.941077:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.941079:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.941083:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.941086:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941098:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.941106:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.941108:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.941110:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.941112:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.941115:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.941118:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.941123:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.941127:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.941132:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.941135:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.941137:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.941139:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1f9 00000080:00010000:2.0:1787827175.941141:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1f9 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.941142:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.941147:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787827175.941156:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.941159:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.941163:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.941164:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.941168:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941173:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.941178:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.941180:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941184:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.941188:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.941190:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.941192:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.941193:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.941194:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.941196:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.941197:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.941200:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.941211:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.941214:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.941218:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941222:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.941227:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.941229:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.941230:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.941232:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.941233:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941238:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.941242:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.941244:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.941246:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.941247:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.941250:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.941252:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.941256:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.941259:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787827175.941266:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787827175.941270:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.941272:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.941273:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1f9 00000080:00010000:2.0:1787827175.941275:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1f9 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.941276:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787827175.941280:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1787827175.941290:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.941292:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.941295:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.941297:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.941300:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941306:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.941310:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.941312:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941320:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.941324:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.941326:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.941327:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.941328:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.941329:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.941331:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.941332:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.941335:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.941343:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.941347:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.941350:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941355:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.941359:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.941361:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.941362:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.941364:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.941366:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941373:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.941377:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.941379:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.941381:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.941382:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.941385:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.941387:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.941390:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.941393:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1787827175.941398:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1787827175.941402:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.941403:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.941405:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1f9 00000080:00010000:2.0:1787827175.941406:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1f9 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.941408:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1787827175.941412:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1787827175.941420:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.941423:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.941426:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.941427:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.941431:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941436:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.941441:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.941443:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941447:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.941451:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.941453:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.941455:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.941456:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.941457:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.941458:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.941459:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.941462:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.941471:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.941475:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.941478:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941483:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.941487:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.941489:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.941490:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.941492:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.941494:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941498:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.941502:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.941504:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.941505:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.941507:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.941509:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.941512:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.941515:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.941518:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1787827175.941526:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1787827175.941530:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.941532:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.941533:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1f9 00000080:00010000:2.0:1787827175.941535:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1f9 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.941536:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1787827175.941540:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1787827175.941546:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.941549:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.941553:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.941554:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.941557:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941562:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.941567:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.941569:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941577:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.941583:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.941585:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.941586:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.941587:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.941589:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.941591:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.941593:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.941597:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1787827175.941603:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000080:00200000:2.0:1787827175.941612:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.941620:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.941626:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941635:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.941642:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.941666:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.941668:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.941671:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.941676:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941685:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.941692:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.941694:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.941697:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.941699:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.941702:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.941705:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.941710:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.941716:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1787827175.941724:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1787827175.941730:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.941733:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.941737:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d1f9 00000080:00010000:2.0:1787827175.941741:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d1f9 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.941756:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1787827175.941762:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae200/0x61d2feb329d0d1f9 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1934 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 0 00000080:00200000:2.0:1787827175.941779:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.941784:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.941788:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.941790:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.941795:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941802:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.941807:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.941809:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941814:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.941819:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.941824:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.941828:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.941830:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.941835:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.941841:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.941845:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.941852:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.941869:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.941875:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.941882:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941888:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.941899:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.941902:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.941904:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.941907:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.941911:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.941925:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.941935:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.941938:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.941941:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.941944:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.941948:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.941953:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.941960:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.941967:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 30 res 8589935617/30 (0 0) 00010000:00010000:2.0:1787827175.941992:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.942015:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.942024:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.942049:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517692160:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.942081:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2b40 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.944302:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517692160:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.944311:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.944319:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.944324:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.944330:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.944336:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.944341:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.944348:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.944352:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.944358:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517692160/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 576/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.944363:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.944369:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.944374:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.944376:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787827175.944399:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.944401:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d207 00000080:00010000:2.0:1787827175.944403:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.944405:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d207 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.944407:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.944412:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.944427:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.944431:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.944438:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.944440:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.944447:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.944453:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.944459:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.944461:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.944466:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.944471:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.944474:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.944476:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.944478:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.944479:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.944481:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.944483:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.944487:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.944507:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.944513:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.944517:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.944522:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.944527:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.944529:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.944531:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.944533:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.944535:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.944539:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.944543:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.944546:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.944548:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.944552:0:21402:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), flags=9 00000080:00200000:2.0:1787827175.944554:0:21402:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), intent=getattr 00000002:00010000:2.0:1787827175.944559:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile0,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.944571:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.944575:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.944580:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.944587:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517692544:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.944615:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2b40 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.945825:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517692544:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.945835:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.945845:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.945851:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.945855:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.945861:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.945866:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.945871:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.945879:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.945884:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517692544/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.945889:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.945895:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.945901:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.945906:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.945911:0:21402:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.945915:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.945919:0:21402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af400/0x61d2feb329d0d215 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xb0788cb3d22a19ab expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.945937:0:21402:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1787827175.945940:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.945954:0:21402:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8): rc = 0 00000080:00200000:2.0:1787827175.945960:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.945964:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d207 00000080:00010000:2.0:1787827175.945972:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.945975:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d207 from it ffffb4b781ec3b20 00010000:00010000:2.0:1787827175.945980:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.945998:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200008:2.0:1787827175.946017:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.946024:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.946031:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.946039:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827175.946046:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827175.946051:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.946054:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.946060:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d207 00000080:00010000:2.0:1787827175.946066:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d207 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.946072:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827175.946081:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787827175.946096:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.946104:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000100:00100000:1.0:1787827175.946108:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517692928:192.168.203.101@tcp:103:ldlm_bl.0 00000008:00010000:2.0:1787827175.946118:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.946121:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.946132:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.946145:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.946164:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.946168:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.946177:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.946187:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.946192:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.946195:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.946198:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.946201:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.946205:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.946208:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.946217:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.946241:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.946247:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.946254:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.946262:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.946268:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.946272:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.946275:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.946279:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.946282:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.946289:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.946299:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.946303:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.946307:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.946310:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.946314:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.946318:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.946327:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.946334:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.946342:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787827175.946348:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.946358:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.946360:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d207 00000080:00010000:2.0:1787827175.946364:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d207 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.946367:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.946374:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.946385:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000100:00100000:1.0:1787827175.946389:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000001:00010000:2.0:1787827175.946390:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.946396:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.946398:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.946406:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.946416:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.946418:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000020:00200000:2.0:1787827175.946424:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.946427:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.946436:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.946439:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524398720 00000100:00100000:1.0:1787827175.946444:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524398720 00000020:00200000:2.0:1787827175.946450:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.946455:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.946457:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.946460:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.946463:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.946466:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.946469:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.946474:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.946490:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000100:00100000:1.0:1787827175.946499:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1189 00000002:00010000:2.0:1787827175.946500:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.946509:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.946512:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87fdb480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:8207:x1874672524398720:12345-192.168.203.101@tcp:104: 00010000:00010000:2.0:1787827175.946520:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.946525:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aee00/0x61d2feb329d0d207 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a1973 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.946532:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.946536:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.946539:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.946542:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.946546:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.946559:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.946568:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000080:00200000:2.0:1787827175.946573:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000100:00100000:1.0:1787827175.946580:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87fdb480 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+6:8207:x1874672524398720:12345-192.168.203.101@tcp:104: Request processed in 68us (159us total) trans 0 rc 0/0 00000080:00200000:2.0:1787827175.946581:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.946586:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.946590:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000100:00100000:1.0:1787827175.946591:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1189 00000080:00200008:2.0:1787827175.946600:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.946611:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.946624:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.946635:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 30 res 8589935617/30 (0 0) 00010000:00010000:2.0:1787827175.946683:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.946691:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.946700:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.946713:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517693056:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.946770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2480 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.946803:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8804c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517692928:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.947025:0:2348:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1787827175.947037:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8804c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517693184:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:1.0:1787827175.947794:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8804c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517693184:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787827175.948417:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8804c3c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517692800:192.168.203.101@tcp:36:createmany.0 00000080:00200000:2.0:1787827175.948463:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:2.0:1787827175.948471:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.948479:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.948487:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.948492:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.948497:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:2.0:1787827175.948504:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:2.0:1787827175.948509:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:2.0:1787827175.948511:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.948519:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.948530:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.948535:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:2.0:1787827175.948539:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.948548:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:2.0:1787827175.948553:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:2.0:1787827175.948560:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.948583:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.948595:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.948603:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.948621:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803e1c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517693312:192.168.203.101@tcp:101:createmany.0 00000100:00100000:2.0:1787827175.948682:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.949173:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517693056:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.949188:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949197:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949201:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949208:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949214:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949219:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949223:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949228:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.949238:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517693056/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 576/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.949248:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.949258:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949267:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.949271:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 0 00000080:00200000:2.0:1787827175.949304:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.949308:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d21c 00000080:00010000:2.0:1787827175.949312:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.949315:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d21c from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.949318:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949327:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.949345:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.949353:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.949365:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.949367:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.949377:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.949393:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.949403:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.949406:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.949414:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.949423:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.949429:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.949433:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.949436:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.949439:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.949442:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.949444:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.949452:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.949476:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.949481:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.949487:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.949492:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.949500:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.949502:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.949504:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.949507:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.949509:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.949513:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.949519:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.949521:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.949523:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.949530:0:21402:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), flags=9 00000080:00200000:2.0:1787827175.949532:0:21402:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), intent=getattr 00000002:00010000:2.0:1787827175.949536:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile0,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.949550:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949555:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949559:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.949568:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517693440:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.949598:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2480 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.949692:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803e1c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517693312:192.168.203.101@tcp:101:createmany.0 00010000:00010000:2.0:1787827175.949702:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949709:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949714:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949719:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949724:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949729:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949733:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949741:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.949760:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803e1c0 x1874672517693312/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.949766:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.949773:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949777:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.949780:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.949794:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:2.0:1787827175.949800:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.949802:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d223 00000080:00010000:2.0:1787827175.949806:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:2.0:1787827175.949808:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d223 from it ffffb4b781c77ba0 00010000:00010000:2.0:1787827175.949810:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.949815:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.949829:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:2.0:1787827175.949833:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.949837:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.949842:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.949845:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.949847:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:2.0:1787827175.949849:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:2.0:1787827175.949851:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:2.0:1787827175.949853:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.949857:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.949861:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.949864:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:2.0:1787827175.949866:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.949870:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1118, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:2.0:1787827175.949872:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:2.0:1787827175.949874:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00200000:2.0:1787827175.949876:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:2.0:1787827175.949879:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1118 00010000:00010000:2.0:1787827175.949887:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.949895:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.949909:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.949913:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787827175.949915:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af000/0x61d2feb329d0d223 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a19ea expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000020:00100000:2.0:1787827175.949927:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000100:00100000:2.0:1787827175.949931:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e8803e1c0 x1874672517693696/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827175.949939:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803e1c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517693696:192.168.203.101@tcp:36:createmany.0 00000100:00100000:2.0:1787827175.949982:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.950586:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517693440:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.950597:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.950607:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.950614:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.950623:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.950632:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.950641:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.950668:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.950681:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.950692:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517693440/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.950699:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.950710:0:2339:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000002:00010000:2.0:1787827175.950710:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.950724:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.950733:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.950759:0:21402:0:(ldlm_lock.c:954:ldlm_lock_decref_and_cancel()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.950768:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.950776:0:21402:0:(ldlm_lock.c:887:ldlm_lock_decref_internal()) ### final decref done on CBPENDING lock ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af200/0x61d2feb329d0d22a lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10000400000000 nid: local remote: 0xb0788cb3d22a1a06 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.950799:0:21402:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to regular blp list, 1 blwis in pool 00000080:00200000:2.0:1787827175.950809:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000100:00100000:1.0:1787827175.950816:0:2340:0:(events.c:340:request_in_callback()) peer: 12345-192.168.203.101@tcp (source: 12345-192.168.203.101@tcp) 00000080:00200000:2.0:1787827175.950835:0:21402:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8): rc = 0 00000080:00200000:2.0:1787827175.950853:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.950862:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d21c 00000080:00010000:2.0:1787827175.950874:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.950880:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d21c from it ffffb4b781ec3b20 00010000:00010000:2.0:1787827175.950885:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.950901:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200008:2.0:1787827175.950925:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.950937:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.950949:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.950957:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 7 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827175.950970:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.950978:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.950984:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.950987:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d21c 00000080:00010000:2.0:1787827175.950998:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d21c from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.951005:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x10420000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827175.951026:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae800/0x61d2feb329d0d21c lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 5 type: IBT gid 0 flags: 0x10420000000000 nid: local remote: 0xb0788cb3d22a19ce expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.951036:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.951041:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.951050:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.951054:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.951063:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.951073:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.951085:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.951090:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.951122:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.951145:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.951154:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.951160:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.951167:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.951171:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.951175:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.951177:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.951184:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.951213:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.951225:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.951236:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.951247:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.951254:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.951258:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.951261:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.951265:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.951269:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.951276:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.951286:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.951291:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.951296:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.951306:0:21402:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), flags=9 00000080:00200000:2.0:1787827175.951315:0:21402:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), intent=getattr 00000002:00010000:2.0:1787827175.951324:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile0,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.951350:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.951358:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.951366:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.951380:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517694080:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.951418:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.952610:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803e1c0 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517693696:192.168.203.101@tcp:36:createmany.0 00000080:00200000:2.0:1787827175.952673:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:2.0:1787827175.952683:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.952691:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.952700:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.952707:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.952711:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:2.0:1787827175.952712:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:2.0:1787827175.952715:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:2.0:1787827175.952717:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.952722:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.952732:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.952735:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:2.0:1787827175.952737:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.952752:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:2.0:1787827175.952756:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:2.0:1787827175.952761:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.952777:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.952781:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.952786:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.952794:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803f840 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517694208:192.168.203.101@tcp:101:createmany.0 00000100:00100000:2.0:1787827175.952823:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.953561:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517694080:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.953569:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.953575:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.953579:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.953592:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.953600:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.953609:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.953616:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.953624:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.953632:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517694080/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.953639:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.953666:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.953675:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.953680:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.953701:0:21402:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8): rc = 0 00000080:00200000:2.0:1787827175.953711:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.953715:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d231 00000080:00010000:2.0:1787827175.953720:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.953722:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d231 from it ffffb4b781ec3b20 00010000:00010000:2.0:1787827175.953724:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.953729:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1787827175.953754:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.953759:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.953766:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.953771:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.953777:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.953781:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.953784:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.953786:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d231 00000080:00010000:2.0:1787827175.953788:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d231 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.953790:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.953796:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.953809:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.953815:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.953825:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.953829:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.953838:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.953848:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.953858:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.953861:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.953867:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803f840 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517694208:192.168.203.101@tcp:101:createmany.0 00010000:00010000:2.0:1787827175.953875:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.953880:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787827175.953887:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00010000:00010000:1.0:1787827175.953891:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827175.953893:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.953899:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00010000:00010000:1.0:1787827175.953902:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1787827175.953903:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.953909:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.953914:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00010000:00010000:1.0:1787827175.953915:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1787827175.953918:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.953927:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1787827175.953930:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.953938:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.953942:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.953947:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827175.953948:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.953957:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000002:00100000:1.0:1787827175.953958:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803f840 x1874672517694208/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00010000:00010000:2.0:1787827175.953970:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000002:00010000:1.0:1787827175.953971:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.953992:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000002:00010000:1.0:1787827175.953993:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827175.954012:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00010000:00010000:1.0:1787827175.954015:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00010000:2.0:1787827175.954016:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00200000:1.0:1787827175.954019:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00010000:2.0:1787827175.954020:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.954026:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.954028:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954033:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.954039:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.954040:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:2.0:1787827175.954043:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.954047:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.954049:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00200000:2.0:1787827175.954050:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00010000:1.0:1787827175.954054:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d238 00000080:00200008:2.0:1787827175.954057:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00010000:1.0:1787827175.954060:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00200000:2.0:1787827175.954062:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000080:00010000:1.0:1787827175.954064:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d238 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.954068:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.954069:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.954077:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.954090:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827175.954100:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.954104:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.954107:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00200000:1.0:1787827175.954109:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000080:00010000:2.0:1787827175.954110:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d231 00000080:00010000:2.0:1787827175.954115:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d231 from it ffffb4b781ec3cb8 00000002:00010000:1.0:1787827175.954117:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.954117:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:1.0:1787827175.954129:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954129:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.954144:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.954145:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000080:00200000:1.0:1787827175.954154:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000001:00010000:2.0:1787827175.954154:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000080:00010000:1.0:1787827175.954161:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000008:00010000:2.0:1787827175.954163:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000080:00010000:1.0:1787827175.954164:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000008:00010000:2.0:1787827175.954167:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00000080:00010000:1.0:1787827175.954168:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:1.0:1787827175.954171:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954174:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.954179:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954188:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.954191:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000020:00200000:2.0:1787827175.954197:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00200000:1.0:1787827175.954200:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00010000:00010000:2.0:1787827175.954201:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.954204:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.954216:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1119, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00010000:00010000:2.0:1787827175.954216:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.954223:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.954230:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000020:00200000:2.0:1787827175.954231:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787827175.954233:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:2.0:1787827175.954235:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000080:00200000:1.0:1787827175.954237:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1119 00000020:00010000:2.0:1787827175.954238:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.954242:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.954243:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.954246:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.954248:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00010000:00010000:1.0:1787827175.954249:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.954253:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1787827175.954261:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.954267:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.954275:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.954282:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954288:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.954292:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.954295:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.954297:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00010000:00010000:1.0:1787827175.954299:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00010000:2.0:1787827175.954300:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.954304:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.954318:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.954324:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869af600/0x61d2feb329d0d238 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a1a53 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.954326:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.954345:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.954348:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000020:00100000:1.0:1787827175.954349:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000080:00200000:2.0:1787827175.954351:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.954355:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000100:00100000:1.0:1787827175.954356:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87fd8f00 x1874672517694464/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000080:00200008:2.0:1787827175.954359:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.954368:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000100:00100000:1.0:1787827175.954375:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fd8f00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517694464:192.168.203.101@tcp:36:createmany.0 00000002:00010000:2.0:1787827175.954375:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.954384:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827175.954391:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827175.954395:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.954398:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.954399:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d231 00000080:00010000:2.0:1787827175.954402:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d231 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.954404:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:2.0:1787827175.954409:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000100:00100000:1.0:1787827175.954414:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000080:00200000:2.0:1787827175.954421:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.954426:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.954432:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.954433:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.954439:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954444:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.954449:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.954451:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954455:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.954461:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.954464:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.954465:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.954467:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.954469:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.954471:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.954473:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.954476:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.954487:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.954490:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.954494:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954499:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.954502:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.954505:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.954507:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.954510:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.954513:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954521:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.954529:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.954532:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.954535:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.954538:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.954542:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.954547:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.954555:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.954561:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.954573:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787827175.954577:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.954580:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.954581:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d231 00000080:00010000:2.0:1787827175.954585:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d231 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.954587:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.954595:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.954605:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.954609:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.954616:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.954618:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.954625:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954632:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.954637:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.954639:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954669:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.954677:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.954681:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.954683:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.954684:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.954686:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.954688:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.954689:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.954693:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.954704:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.954708:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.954713:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954718:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.954722:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.954724:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.954726:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.954728:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.954729:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954737:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.954753:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.954756:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.954759:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.954760:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.954763:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.954766:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.954771:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.954776:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.954781:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.954784:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.954786:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.954788:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d231 00000080:00010000:2.0:1787827175.954790:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d231 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.954791:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.954796:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787827175.954805:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.954808:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.954812:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.954813:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.954818:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954823:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.954827:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.954829:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954833:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.954837:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.954840:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.954841:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.954842:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.954844:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.954845:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.954846:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.954850:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.954860:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.954864:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.954867:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954872:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.954876:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.954878:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.954879:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.954881:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.954883:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954887:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.954892:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.954893:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.954895:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.954897:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.954899:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.954902:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.954905:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.954908:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787827175.954916:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787827175.954920:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.954921:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.954923:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d231 00000080:00010000:2.0:1787827175.954924:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d231 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.954926:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787827175.954930:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1787827175.954935:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.954937:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.954941:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.954942:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.954946:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954951:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.954956:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.954957:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.954965:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.954969:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.954971:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.954972:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.954974:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.954975:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.954976:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.954978:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.954981:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.954989:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.954992:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.954995:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.955000:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.955004:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.955006:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.955008:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.955009:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.955011:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.955019:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.955027:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.955030:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.955033:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.955036:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.955040:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.955044:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.955050:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.955057:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1787827175.955064:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1787827175.955070:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.955074:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.955077:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d231 00000080:00010000:2.0:1787827175.955080:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d231 from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.955083:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1787827175.955090:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1787827175.955106:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.955110:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.955117:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.955119:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.955126:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.955135:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.955143:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.955146:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.955154:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.955161:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.955165:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.955166:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.955168:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.955170:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.955172:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.955174:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.955179:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.955192:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.955196:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.955201:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.955209:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.955216:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.955220:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.955223:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.955227:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.955231:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.955239:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.955254:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.955260:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.955264:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.955267:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.955273:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.955278:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.955287:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00000100:00100000:1.0:1787827175.955289:0:15045:0:(service.c:155:ptlrpc_grow_req_bufs()) ldlm_cbd: allocate 1 new 8192-byte reqbufs (0/2 left), rc = 0 00010000:00010000:2.0:1787827175.955295:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000100:00100000:1.0:1787827175.955299:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524399104 00000100:00100000:1.0:1787827175.955305:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524399104 00010000:00010000:2.0:1787827175.955315:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1787827175.955322:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.955327:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.955329:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d231 00000080:00010000:2.0:1787827175.955334:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d231 from it ffffb4b781ec3cb8 00000100:00100000:1.0:1787827175.955338:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1192 00010000:00010000:2.0:1787827175.955340:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000100:00100000:1.0:1787827175.955348:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e87ffc050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524399104:12345-192.168.203.101@tcp:104: 00010000:00010000:2.0:1787827175.955357:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.955361:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aea00/0x61d2feb329d0d231 lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a1a37 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1787827175.955376:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.955383:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.955391:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.955393:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:1.0:1787827175.955402:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00010000:00010000:2.0:1787827175.955402:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.955410:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e87ffc050 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:8207:x1874672524399104:12345-192.168.203.101@tcp:104: Request processed in 65us (169us total) trans 0 rc 0/0 00010000:00010000:2.0:1787827175.955422:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.955423:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1192 00000020:00200000:2.0:1787827175.955448:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.955454:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.955483:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.955497:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.955504:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.955508:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.955511:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.955514:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.955519:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.955522:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.955529:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.955555:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.955565:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.955574:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.955590:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.955597:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.955602:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.955605:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.955609:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.955612:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.955624:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.955634:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.955638:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.955642:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.955674:0:21402:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), flags=9 00000080:00200000:2.0:1787827175.955679:0:21402:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), intent=getattr 00000002:00010000:2.0:1787827175.955688:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile0,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.955711:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.955717:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.955724:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.955737:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517694720:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.955792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.957080:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fd8f00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517694464:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.957116:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.957123:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.957130:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.957136:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.957140:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.957143:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.957145:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.957147:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.957149:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.957154:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.957161:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.957164:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.957166:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.957171:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.957174:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.957178:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.957191:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.957195:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.957200:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.957207:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fd8780 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517694848:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.957235:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.957862:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517694720:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.957871:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.957878:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.957886:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.957892:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.957898:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.957903:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.957907:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.957911:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.957917:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517694720/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.957922:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.957928:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.957936:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.957939:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.957962:0:21402:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8): rc = 0 00000080:00200000:2.0:1787827175.957969:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.957971:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d23f 00000080:00010000:2.0:1787827175.957975:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.957977:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d23f from it ffffb4b781ec3b20 00010000:00010000:2.0:1787827175.957979:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.957983:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1787827175.957996:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.958000:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.958006:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.958010:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.958015:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.958018:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.958020:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.958022:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d23f 00000080:00010000:2.0:1787827175.958024:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d23f from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.958026:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.958031:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.958038:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.958042:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.958049:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.958051:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.958057:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.958063:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.958069:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.958071:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.958084:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.958092:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.958095:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.958097:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.958099:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.958101:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.958103:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.958105:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.958109:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.958124:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.958129:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.958133:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.958138:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.958142:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.958145:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.958146:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.958148:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.958150:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.958154:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.958159:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.958162:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.958164:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.958166:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.958169:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.958172:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.958176:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.958180:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827175.958185:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.958189:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.958191:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.958192:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d23f 00000080:00010000:2.0:1787827175.958194:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d23f from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.958196:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827175.958200:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827175.958206:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.958209:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.958212:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.958214:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.958218:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.958224:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.958230:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.958232:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.958238:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.958244:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.958248:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.958250:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.958251:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.958253:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.958254:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.958256:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.958262:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.958273:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000100:00100000:1.0:1787827175.958277:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fd8780 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517694848:192.168.203.101@tcp:101:createmany.0 00000002:00010000:2.0:1787827175.958278:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.958289:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.958292:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.958300:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.958309:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00010000:00010000:1.0:1787827175.958310:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1787827175.958314:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.958319:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00010000:00010000:1.0:1787827175.958324:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00010000:2.0:1787827175.958325:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.958330:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.958338:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.958356:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.958372:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827175.958374:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.958382:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00010000:00010000:1.0:1787827175.958384:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827175.958387:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.958395:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00010000:00010000:1.0:1787827175.958396:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.958403:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200008:2.0:1787827175.958404:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000002:00100000:1.0:1787827175.958416:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87fd8780 x1874672517694848/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000080:00200000:2.0:1787827175.958416:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:1.0:1787827175.958426:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.958428:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00000002:00010000:1.0:1787827175.958434:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.958437:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.958441:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:1.0:1787827175.958448:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1787827175.958452:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827175.958461:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00200000:1.0:1787827175.958466:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00010000:2.0:1787827175.958466:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.958473:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d23f 00000080:00200000:1.0:1787827175.958477:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.958479:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d23f from it ffffb4b781ec3cb8 00000080:00010000:1.0:1787827175.958481:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d246 00010000:00010000:2.0:1787827175.958484:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00010000:1.0:1787827175.958487:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.958490:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d246 from it ffffb4b781c77ba0 00010000:00010000:1.0:1787827175.958494:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.958504:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:1.0:1787827175.958507:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.958534:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.958537:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.958543:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.958545:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00000080:00200000:1.0:1787827175.958546:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00010000:00010000:2.0:1787827175.958551:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000002:00010000:1.0:1787827175.958556:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.958558:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.958567:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.958569:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.958573:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.958577:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.958578:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.958582:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.958586:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.958592:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.958593:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.958596:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00010000:00010000:1.0:1787827175.958597:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000020:00010000:2.0:1787827175.958598:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.958607:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787827175.958609:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.958618:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.958622:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.958627:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00000080:00200000:2.0:1787827175.958630:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00010000:00010000:1.0:1787827175.958631:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000002:00010000:2.0:1787827175.958640:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.958642:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.958682:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00010000:00010000:2.0:1787827175.958687:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.958689:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.958695:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00010000:00010000:2.0:1787827175.958698:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.958704:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1120, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000080:00200000:2.0:1787827175.958704:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00200000:1.0:1787827175.958711:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00010000:2.0:1787827175.958712:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.958715:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00200000:1.0:1787827175.958716:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000080:00010000:2.0:1787827175.958719:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00000080:00200000:1.0:1787827175.958720:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00010000:00010000:2.0:1787827175.958723:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.958725:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1120 00010000:00010000:2.0:1787827175.958735:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.958754:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.958755:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.958760:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.958765:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.958769:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.958775:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.958779:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00010000:00010000:1.0:1787827175.958781:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00000002:00010000:2.0:1787827175.958785:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.958791:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.958812:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000080:00200000:2.0:1787827175.958817:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.958820:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.958822:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d23f 00000080:00010000:2.0:1787827175.958824:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d23f from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.958827:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.958832:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:1.0:1787827175.958835:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.958840:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00010000:00010000:1.0:1787827175.958842:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00000001:00010000:2.0:1787827175.958845:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00010000:00010000:1.0:1787827175.958846:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8c400/0x61d2feb329d0d246 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a1aa0 expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000008:00010000:2.0:1787827175.958852:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.958855:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00000020:00100000:1.0:1787827175.958859:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00010000:00010000:2.0:1787827175.958861:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.958866:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87fd8780 x1874672517695104/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827175.958873:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.958879:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000100:00100000:1.0:1787827175.958882:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fd8780 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517695104:192.168.203.101@tcp:36:createmany.0 00010000:00010000:2.0:1787827175.958884:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.958896:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.958902:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.958905:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.958907:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.958909:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.958910:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.958912:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.958914:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.958918:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:1.0:1787827175.958929:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000080:00200000:2.0:1787827175.958930:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.958938:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.958943:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.958949:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.958955:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.958959:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.958962:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.958965:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.958968:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.958981:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.958992:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.958996:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.959000:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.959003:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.959006:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.959009:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.959015:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.959019:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.959025:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.959029:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.959032:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.959033:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d23f 00000080:00010000:2.0:1787827175.959036:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d23f from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.959038:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.959042:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787827175.959052:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.959054:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.959058:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.959060:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.959064:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959069:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.959075:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.959077:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959081:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.959085:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.959087:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.959089:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.959090:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.959091:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.959093:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.959094:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.959098:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.959108:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.959113:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.959118:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959126:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.959132:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.959135:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.959137:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.959141:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.959143:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959150:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.959157:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.959161:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.959164:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.959166:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.959170:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.959174:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.959181:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.959186:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787827175.959197:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787827175.959203:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.959206:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.959208:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d23f 00000080:00010000:2.0:1787827175.959211:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d23f from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.959214:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787827175.959220:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1787827175.959228:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.959232:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.959238:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.959239:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.959246:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959253:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.959260:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.959263:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959274:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.959281:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.959284:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.959287:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.959289:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.959292:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.959294:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.959296:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.959302:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.959315:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.959321:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.959327:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959335:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.959341:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.959345:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.959347:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.959350:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.959353:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959364:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.959371:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.959375:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.959378:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.959380:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.959385:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.959389:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.959396:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.959401:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1787827175.959410:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1787827175.959414:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.959417:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.959419:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d23f 00000080:00010000:2.0:1787827175.959421:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d23f from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.959423:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1787827175.959427:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1787827175.959437:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.959440:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.959444:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.959445:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.959450:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959455:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.959460:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.959462:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959467:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.959473:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.959477:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.959479:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.959481:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.959483:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.959486:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.959488:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.959494:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.959509:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.959516:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.959523:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959529:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.959533:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.959536:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.959537:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.959539:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.959541:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959546:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.959551:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.959553:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.959555:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.959557:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.959560:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.959563:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.959567:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.959570:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1787827175.959578:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1787827175.959582:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.959583:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.959585:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d23f 00000080:00010000:2.0:1787827175.959587:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d23f from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.959588:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1787827175.959593:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1787827175.959599:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.959603:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.959608:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.959610:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.959617:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959626:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.959634:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.959637:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959673:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.959681:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.959686:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.959688:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.959690:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.959692:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.959694:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.959699:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.959705:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.959718:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.959723:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.959728:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959734:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.959738:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000100:00100000:1.0:1787827175.959750:0:5567:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524399232 00000080:00010000:2.0:1787827175.959753:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000100:00100000:1.0:1787827175.959757:0:5567:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524399232 00000080:00010000:2.0:1787827175.959757:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.959762:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.959765:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959773:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.959780:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.959782:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.959785:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.959787:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.959792:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000100:00100000:1.0:1787827175.959797:0:5567:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1193 00000080:00200000:2.0:1787827175.959799:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000100:00100000:1.0:1787827175.959806:0:5567:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804d680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5840:x1874672524399232:12345-192.168.203.101@tcp:104: 00000002:00010000:2.0:1787827175.959807:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.959815:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1787827175.959821:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:1.0:1787827175.959823:0:5567:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1787827175.959826:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.959833:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.959835:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d23f 00000080:00010000:2.0:1787827175.959837:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d23f from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.959839:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10420000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1787827175.959844:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae600/0x61d2feb329d0d23f lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10420000000000 nid: local remote: 0xb0788cb3d22a1a84 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1787827175.959853:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.959857:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.959863:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.959864:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:1.0:1787827175.959867:0:5567:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00010000:00010000:2.0:1787827175.959870:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.959875:0:5567:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804d680 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_000:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5840:x1874672524399232:12345-192.168.203.101@tcp:104: Request processed in 70us (178us total) trans 0 rc 0/0 00010000:00010000:2.0:1787827175.959884:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.959890:0:5567:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1193 00000020:00200000:2.0:1787827175.959898:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.959902:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.959915:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.959933:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.959937:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.959942:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.959945:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.959948:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.959955:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.959961:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.959969:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.959985:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.959991:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.959998:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.960005:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.960022:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.960026:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.960028:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.960031:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.960033:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.960045:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.960054:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.960056:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.960060:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.960065:0:21402:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), flags=9 00000080:00200000:2.0:1787827175.960068:0:21402:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile0, dir=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), intent=getattr 00000002:00010000:2.0:1787827175.960075:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile0,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.960096:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.960102:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.960111:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.960127:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517695360:192.168.203.101@tcp:101:statmany.0 00000100:00100000:2.0:1787827175.960167:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.961556:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fd8780 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517695104:192.168.203.101@tcp:36:createmany.0 00000080:00200000:1.0:1787827175.961600:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:1.0:1787827175.961611:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.961621:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.961630:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.961638:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.961643:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.961675:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:1.0:1787827175.961681:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:1.0:1787827175.961685:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.961693:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.961702:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.961706:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:1.0:1787827175.961710:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.961717:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:1.0:1787827175.961721:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:1.0:1787827175.961728:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:1.0:1787827175.961764:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.961771:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.961778:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:1.0:1787827175.961790:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fdad00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517695488:192.168.203.101@tcp:101:createmany.0 00000100:00100000:1.0:1787827175.961827:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000100:00100000:2.0:1787827175.962385:0:21402:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517695360:192.168.203.101@tcp:101:statmany.0 00010000:00010000:2.0:1787827175.962397:0:21402:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.962407:0:21402:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.962413:0:21402:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.962423:0:21402:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.962433:0:21402:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.962443:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.962450:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.962463:0:21402:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00100000:2.0:1787827175.962473:0:21402:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e8803fc00 x1874672517695360/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'statmany.0' uid:0 gid:0 projid:0 00000002:00010000:2.0:1787827175.962483:0:21402:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.962494:0:21402:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.962503:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 4 res 8589935617/30 (0 0) 00000080:00200000:2.0:1787827175.962507:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00000080:00200000:2.0:1787827175.962533:0:21402:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8): rc = 0 00000080:00200000:2.0:1787827175.962542:0:21402:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00010000:2.0:1787827175.962546:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d24d 00000080:00010000:2.0:1787827175.962553:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.962556:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d24d from it ffffb4b781ec3b20 00010000:00010000:2.0:1787827175.962559:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.962567:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200008:2.0:1787827175.962587:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.962593:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.962601:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.962608:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.962616:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.962624:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.962628:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.962631:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d24d 00000080:00010000:2.0:1787827175.962635:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d24d from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.962638:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.962663:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.962678:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.962684:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.962695:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.962697:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.962707:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.962717:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.962726:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.962729:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.962738:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.962757:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.962762:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.962765:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.962768:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.962771:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.962775:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.962777:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.962784:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.962804:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.962810:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.962817:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.962825:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.962832:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.962836:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.962839:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.962842:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.962845:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.962854:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.962863:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.962867:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.962871:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.962874:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.962879:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.962885:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.962892:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.962899:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827175.962908:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:2.0:1787827175.962915:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.962919:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.962922:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d24d 00000080:00010000:2.0:1787827175.962926:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d24d from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.962929:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827175.962938:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827175.962955:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.962960:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000100:00100000:1.0:1787827175.962967:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fdad00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517695488:192.168.203.101@tcp:101:createmany.0 00000008:00010000:2.0:1787827175.962968:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.962973:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:1.0:1787827175.962978:0:21400:0:(ldlm_request.c:759:ldlm_cli_enqueue_fini()) ### server returned different mode PR ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 4/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.962984:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.962988:0:21400:0:(ldlm_request.c:774:ldlm_cli_enqueue_fini()) ### client-side enqueue, new resource ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.963003:0:21400:0:(ldlm_request.c:787:ldlm_cli_enqueue_fini()) ### client-side enqueue, new policy data ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 4/1,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.963005:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.963012:0:21400:0:(ldlm_lock.c:1072:ldlm_granted_list_add_lock()) ### About to add lock: ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787827175.963025:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:1.0:1787827175.963026:0:21400:0:(ldlm_request.c:841:ldlm_cli_enqueue_fini()) ### client-side enqueue END ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.963030:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.963045:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.963048:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.963066:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 5/2,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00200000:2.0:1787827175.963068:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.963074:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00010000:00010000:1.0:1787827175.963078:0:21400:0:(ldlm_lock.c:919:ldlm_lock_decref_internal()) ### do not add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 4/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000020:00010000:2.0:1787827175.963079:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.963084:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.963091:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.963096:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000002:00100000:1.0:1787827175.963100:0:21400:0:(mdc_locks.c:835:mdc_finish_enqueue()) @@@ op=0x8 disposition=0xb, status=0 req@ffff8b2e87fdad00 x1874672517695488/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/640 e 0 to 0 dl 1787827191 ref 1 fl Complete:RQU/600/0 rc 0/0 job:'createmany.0' uid:0 gid:0 projid:0 00000020:00010000:2.0:1787827175.963101:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.963111:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000002:00010000:1.0:1787827175.963115:0:21400:0:(mdc_locks.c:939:mdc_finish_enqueue()) ### layout lock returned by: getattr, lvb_len: 56 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 0 lru_score: 0 lru_type: 0 00000080:00200000:2.0:1787827175.963127:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:1.0:1787827175.963128:0:21400:0:(mdc_locks.c:1359:mdc_finish_intent_lock()) ### matching against this ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00000002:00010000:2.0:1787827175.963135:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.963140:0:21400:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e88a6a800 type 13 mode 4 res 8589935617/30 (0 0) 00010000:00010000:2.0:1787827175.963142:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.963149:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 -1: rc 0 00010000:00010000:2.0:1787827175.963152:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.963157:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.963160:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.963161:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.963164:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.963166:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963171:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.963173:0:21400:0:(namei.c:147:ll_iget()) got inode: [0x200000401:0x1e:0x0](ffff8b2e98e0cb08): rc = 0 00000080:00200000:2.0:1787827175.963177:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.963180:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.963184:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:1.0:1787827175.963186:0:21400:0:(file.c:6854:ll_layout_conf()) [0x200000401:0x1e:0x0]: layout version change: 0 -> 0 00000080:00200000:2.0:1787827175.963186:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.963192:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00010000:1.0:1787827175.963193:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) for lock 0x61d2feb329d0d254 00000080:00200000:2.0:1787827175.963200:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000080:00010000:1.0:1787827175.963202:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0cb08) 00000080:00010000:1.0:1787827175.963206:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d254 from it ffffb4b781c77ba0 00000002:00010000:2.0:1787827175.963207:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.963209:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.963213:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00010000:00010000:1.0:1787827175.963234:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.963238:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:2.0:1787827175.963245:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.963249:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.963250:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d24d 00000080:00010000:2.0:1787827175.963253:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d24d from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.963255:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 3 lru_type: 0 00000080:00200000:1.0:1787827175.963259:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00010000:00010000:2.0:1787827175.963261:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000002:00010000:1.0:1787827175.963266:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00000080:00200000:2.0:1787827175.963271:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00010000:00010000:1.0:1787827175.963274:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000001:00010000:2.0:1787827175.963276:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.963284:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.963286:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.963291:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.963296:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963300:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.963302:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:1.0:1787827175.963309:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:1.0:1787827175.963314:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000020:00200000:2.0:1787827175.963314:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00000080:00010000:1.0:1787827175.963317:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bc0 00010000:00010000:2.0:1787827175.963319:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.963321:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.963337:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963344:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000080:00200000:1.0:1787827175.963348:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:1.0:1787827175.963353:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000020:00200000:2.0:1787827175.963354:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:1.0:1787827175.963356:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.963361:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.963365:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000080:00200000:1.0:1787827175.963366:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile1121, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=1536 00000020:00010000:2.0:1787827175.963369:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000080:00200000:1.0:1787827175.963373:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000020:00010000:2.0:1787827175.963375:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000080:00200000:1.0:1787827175.963381:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), inode mode 81a4 mask 6 00000020:00010000:2.0:1787827175.963382:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.963388:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:1.0:1787827175.963389:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 3 00000080:00200000:1.0:1787827175.963396:0:21400:0:(namei.c:2195:ll_link()) VFS Op: inode=[0x200000401:0x1e:0x0](ffff8b2e98e0cb08), dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), target=lockfile1121 00000080:00200000:2.0:1787827175.963396:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1787827175.963413:0:21400:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.963416:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.963425:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:1.0:1787827175.963430:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d200/0x61d2feb329d0938c lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d228d91e expref: -99 pid: 21382 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963432:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963453:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.963457:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.963460:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.963462:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.963465:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.963467:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.963471:0:21400:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827175.963484:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:1.0:1787827175.963485:0:21400:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:1.0:1787827175.963490:0:21400:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b8d000/0x61d2feb329d0d254 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x4829400000020 nid: local remote: 0xb0788cb3d22a1aed expref: -99 pid: 21400 timeout: 0 lvb_type: 3 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827175.963494:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.963500:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.963502:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.963504:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000020:00100000:1.0:1787827175.963507:0:21400:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 36, max 7 00000080:00200008:2.0:1787827175.963508:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.963516:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000100:00100000:1.0:1787827175.963517:0:21400:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e87fdad00 x1874672517695744/t0(0) o36->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 504/440 e 0 to 0 dl 0 ref 1 fl New:QU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000002:00010000:2.0:1787827175.963522:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.963530:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.963538:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00000100:00100000:1.0:1787827175.963540:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fdad00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517695744:192.168.203.101@tcp:36:createmany.0 00000080:00200000:2.0:1787827175.963550:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.963553:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.963555:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d24d 00000080:00010000:2.0:1787827175.963557:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d24d from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.963559:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 4 lru_type: 0 00010000:00010000:2.0:1787827175.963564:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.963575:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.963578:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.963584:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000100:00100000:1.0:1787827175.963587:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7900 going to sleep for 11 seconds 00000008:00010000:2.0:1787827175.963587:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.963596:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963603:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.963608:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.963610:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963615:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.963620:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.963623:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.963625:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.963627:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.963629:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.963631:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.963632:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.963636:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.963669:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.963677:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.963684:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963693:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.963699:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.963703:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.963705:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.963709:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.963712:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963719:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.963726:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.963730:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.963733:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.963736:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.963752:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.963756:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.963762:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.963767:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.963776:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00000080:00200000:2.0:1787827175.963779:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.963782:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.963783:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d24d 00000080:00010000:2.0:1787827175.963786:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d24d from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.963788:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 5 lru_type: 0 00010000:00010000:2.0:1787827175.963792:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787827175.963799:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.963802:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.963806:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.963807:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.963811:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963816:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.963821:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.963822:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963831:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.963835:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.963837:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.963839:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.963840:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.963841:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.963843:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.963844:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.963848:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.963858:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.963862:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.963865:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963870:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.963874:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.963876:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.963877:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.963879:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.963881:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963888:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.963893:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.963894:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.963896:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.963898:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.963900:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.963903:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.963907:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.963910:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787827175.963914:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00000080:00200000:2.0:1787827175.963918:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.963920:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.963922:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d24d 00000080:00010000:2.0:1787827175.963923:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d24d from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.963925:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 6 lru_type: 0 00010000:00010000:2.0:1787827175.963929:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1787827175.963937:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.963939:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.963943:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.963944:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.963947:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963952:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.963957:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.963958:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.963963:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.963967:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.963969:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.963971:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.963972:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.963974:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.963975:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.963977:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.963980:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.963990:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.963993:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.963997:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964002:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.964005:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.964007:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.964009:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.964010:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.964012:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964016:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.964021:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.964023:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.964025:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.964026:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.964028:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.964031:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.964034:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.964037:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1787827175.964045:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:2.0:1787827175.964048:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.964050:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.964051:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d24d 00000080:00010000:2.0:1787827175.964053:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d24d from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.964055:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00010000:00010000:2.0:1787827175.964058:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1787827175.964063:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.964066:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.964069:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.964070:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.964073:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964078:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.964083:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.964084:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964092:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.964097:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.964098:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.964100:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.964101:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.964102:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.964104:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.964105:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.964108:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.964116:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.964119:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.964123:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964127:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.964131:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.964133:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.964134:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.964136:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.964138:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964146:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.964151:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.964152:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.964154:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.964155:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.964158:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.964160:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.964164:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.964167:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1787827175.964172:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00000080:00200000:2.0:1787827175.964176:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.964178:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.964180:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d24d 00000080:00010000:2.0:1787827175.964181:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d24d from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.964183:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 8 lru_type: 0 00010000:00010000:2.0:1787827175.964187:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1787827175.964195:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.964197:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.964201:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.964202:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.964205:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964211:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.964215:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.964216:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964220:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.964225:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.964227:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.964228:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.964230:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.964231:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.964232:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.964234:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.964237:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.964246:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.964250:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.964253:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964258:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.964262:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.964264:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.964265:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.964267:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.964268:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964273:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.964277:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.964279:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.964281:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.964283:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.964287:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.964290:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.964296:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.964302:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1787827175.964315:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00000080:00200000:2.0:1787827175.964320:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.964324:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.964326:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d24d 00000080:00010000:2.0:1787827175.964329:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d24d from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.964332:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 9 lru_type: 0 00010000:00010000:2.0:1787827175.964340:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 0 00000080:00200000:2.0:1787827175.964349:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.964352:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.964359:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.964362:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.964368:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964377:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.964386:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.964390:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964404:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.964410:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.964414:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.964416:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.964421:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.964424:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.964428:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.964429:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.964435:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827175.964447:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000002:00010000:2.0:1787827175.964452:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.964457:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964463:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000100:00100000:1.0:1787827175.964464:0:15045:0:(service.c:2301:ptlrpc_server_handle_req_in()) unwrap req x1874672524399360 00000080:00200000:2.0:1787827175.964470:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000100:00100000:1.0:1787827175.964472:0:15045:0:(service.c:2366:ptlrpc_server_handle_req_in()) got req x1874672524399360 00000080:00010000:2.0:1787827175.964474:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.964477:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.964480:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.964482:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964490:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.964495:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.964497:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.964499:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.964501:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.964504:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.964508:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.964513:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.964516:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 0 00010000:00010000:2.0:1787827175.964522:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 0 00000100:00100000:1.0:1787827175.964527:0:15045:0:(nrs_fifo.c:152:nrs_fifo_req_get()) NRS start fifo request from 12345-192.168.203.101@tcp, seq: 1194 00000080:00200000:2.0:1787827175.964529:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000401:0x20:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.964533:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) 00000080:00010000:2.0:1787827175.964535:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000401:0x1e:0x0](ffff8b2e98e0e3c8) for lock 0x61d2feb329d0d24d 00000080:00010000:2.0:1787827175.964538:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d0d24d from it ffffb4b781ec3cb8 00010000:00010000:2.0:1787827175.964541:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/1,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x10020000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 10 lru_type: 0 00000100:00100000:1.0:1787827175.964543:0:15045:0:(service.c:2552:ptlrpc_server_handle_request()) Handling RPC req@ffff8b2e8804c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524399360:12345-192.168.203.101@tcp:104: 00010000:00010000:2.0:1787827175.964551:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1787827175.964560:0:15045:0:(ldlm_lockd.c:2506:ldlm_callback_handler()) ### blocking ast ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 3/0,0 mode: PR/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x0 rrc: 4 type: IBT gid 0 flags: 0x420000000000 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00000080:00200000:2.0:1787827175.964562:0:21402:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 restore needed 0 00000001:00010000:2.0:1787827175.964569:0:21402:0:(glimpse.c:73:cl_glimpse_lock()) Glimpsing inode [0x200000401:0x1e:0x0] 00000008:00010000:2.0:1787827175.964575:0:21402:0:(lustre_osc.h:162:osc_enq2ldlm_flags()) flags: a 00000008:00010000:2.0:1787827175.964577:0:21402:0:(osc_lock.c:1320:osc_lock_init()) ### lock ffff8b2e88ae25b0, osc lock ffff8b2e90e62ea0, flags 0x400001002 00010000:00010000:2.0:1787827175.964584:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 3 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964592:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 18446744073709551615) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 2/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.964597:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 1 items remaining. 00010000:00010000:2.0:1787827175.964599:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 5/1,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x810020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964604:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000d2d4cdd8/0x61d2feb329d09d33 lrc: 4/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d22918e9 expref: -99 pid: 21402 timeout: 0 lvb_type: 1 lru_score: 254 lru_type: 1 00000020:00200000:2.0:1787827175.964609:0:21402:0:(cl_io.c:1853:__cl_sync_io_note()) Noting completion of 1 items, 0 items remaining. 00000080:00200000:2.0:1787827175.964611:0:21402:0:(file.c:1775:ll_merge_attr_nolock()) [0x200000401:0x1e:0x0] updating i_size 0 i_blocks 1 00000020:00010000:2.0:1787827175.964613:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) release lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_release():245 00000020:00010000:2.0:1787827175.964614:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.964616:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) cancel lock: ffff8b2e88ae25b0 (ffff8b2e8579aea8/1) at cl_lock_cancel():142 00000020:00010000:2.0:1787827175.964618:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e84aa73d8 (ffff8b2e8579a518/0) at cl_lock_fini():72 00000020:00010000:2.0:1787827175.964619:0:21402:0:(cl_lock.c:36:__cl_lock_trace()) destroy lock: ffff8b2e88ae25b0 (ffff8b2e8579a518/1) at cl_lock_fini():72 00000080:00200000:2.0:1787827175.964624:0:21402:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 0/1, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1787827175.964625:0:15045:0:(ldlm_lockd.c:2189:__ldlm_bl_to_thread()) added 1/1 locks to priority blp list, 1 blwis in pool 00000100:00100000:1.0:1787827175.964637:0:15045:0:(service.c:2610:ptlrpc_server_handle_request()) Handled RPC req@ffff8b2e8804c000 pname:cluuid+ref:pid:xid:nid:opc:job ldlm_cb00_002:8e7ce795-6dad-45e6-b76e-c5bf3324e96e+5:5841:x1874672524399360:12345-192.168.203.101@tcp:104: Request processed in 99us (216us total) trans 0 rc 0/0 00000080:00200000:2.0:1787827175.964642:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88),name=/ 00000100:00100000:1.0:1787827175.964671:0:15045:0:(nrs_fifo.c:211:nrs_fifo_req_stop()) NRS stop fifo request from 12345-192.168.203.101@tcp, seq: 1194 00000002:00010000:2.0:1787827175.964678:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.964688:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964709:0:21402:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.964728:0:21402:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.964732:0:21402:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) 00000080:00010000:2.0:1787827175.964736:0:21402:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1ba88) for lock 0x61d2feb329d09d09 00000080:00010000:2.0:1787827175.964755:0:21402:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09d09 from it ffffb4b781ec3b58 00010000:00010000:2.0:1787827175.964759:0:21402:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964765:0:21402:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8dc00/0x61d2feb329d09d09 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d229184f expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.964771:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1ba88), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.964773:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=25 00000080:00200000:2.0:1787827175.964775:0:21402:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98e0c2c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.964777:0:21402:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockfile0, flags=9 00000080:00200008:2.0:1787827175.964781:0:21402:0:(file.c:6313:ll_getattr_dentry()) START file lockfile0:[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8), request_mask 2047, flags 0, foreign 0 00000080:00200000:2.0:1787827175.964784:0:21402:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000401:0x1e:0x0](ffff8b2e98e0e3c8),name=lockfile0 00000002:00010000:2.0:1787827175.964790:0:21402:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000401:0x1e:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.964795:0:21402:0:(ldlm_lock.c:1539:ldlm_lock_match_with_skip()) ### not matched ns ffff8b2e87368400 type 13 mode 30 res 8589935617/30 (0 0) 00010000:00010000:1.0:1787827175.964799:0:5578:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869ae000/0x61d2feb329d09e6e lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1e:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x20000000000 nid: local remote: 0xb0788cb3d2291ed1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.964816:0:21402:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0d25b lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.964827:0:5578:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 2/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00010000:00010000:2.0:1787827175.964830:0:21402:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0d25b lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.964837:0:5578:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787827175.964846:0:21402:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869aec00/0x61d2feb329d0d25b lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21402 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:1.0:1787827175.964856:0:5578:0:(ldlm_lockd.c:1933:ldlm_handle_bl_callback()) ### client blocking callback handler END ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 1/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00000100:00100000:2.0:1787827175.964859:0:21402:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803fc00 pname:cluuid:pid:xid:nid:opc:job statmany:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:21402:1874672517695872:192.168.203.101@tcp:101:statmany.0 00010000:00010000:1.0:1787827175.964868:0:5578:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e869af000/0x61d2feb329d0d24d lrc: 0/0,0 mode: --/PR res: [0x200000401:0x1e:0x0].0x0 bits 0x1b/0x1b rrc: 3 type: IBT gid 0 flags: 0x4c29400000020 nid: local remote: 0xb0788cb3d22a1ad1 expref: -99 pid: 21402 timeout: 0 lvb_type: 3 lru_score: 11 lru_type: 1 00010000:00010000:1.0:1787827175.964878:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.964880:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827175.964882:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827175.964904:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 11 seconds 00000100:00100000:1.0:1787827175.965882:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fdb480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517696000:192.168.203.101@tcp:103:ldlm_bl.0 00000100:00100000:2.0:1787827175.966441:0:21400:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fdad00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517695744:192.168.203.101@tcp:36:createmany.0 00000080:00200000:2.0:1787827175.966487:0:21400:0:(file.c:6208:ll_inode_revalidate()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08),name=/ 00000002:00010000:2.0:1787827175.966495:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: ,[0x200000007:0x1:0x0]) in obj [0x200000007:0x1:0x0], intent: getattr flags 0 00010000:00010000:2.0:1787827175.966503:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 4 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.966510:0:21400:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 0) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.966515:0:21400:0:(namei.c:952:ll_intent_lock()) intent lock 8 on i1 [0x200000007:0x1:0x0] suppgids 0 0: rc 1 00000080:00010000:2.0:1787827175.966518:0:21400:0:(dcache.c:255:ll_lookup_finish_locks()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) 00000080:00010000:2.0:1787827175.966519:0:21400:0:(llite_internal.h:1989:ll_set_lock_data()) setting l_data to inode [0x200000007:0x1:0x0](ffff8b2e98d1aa08) for lock 0x61d2feb329d09434 00000080:00010000:2.0:1787827175.966522:0:21400:0:(dcache.c:152:ll_intent_drop_lock()) releasing lock with cookie 0x61d2feb329d09434 from it ffffb4b781c77bd8 00010000:00010000:2.0:1787827175.966524:0:21400:0:(ldlm_lock.c:829:ldlm_lock_decref_internal_nolock()) ### ldlm_lock_decref(PR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 3/1,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x10000000000000 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00010000:00010000:2.0:1787827175.966530:0:21400:0:(ldlm_lock.c:912:ldlm_lock_decref_internal()) ### add lock into lru list ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386400/0x61d2feb329d09434 lrc: 2/0,0 mode: PR/PR res: [0x200000007:0x1:0x0].0x0 bits 0x13/0x0 rrc: 3 type: IBT gid 0 flags: 0x0 nid: local remote: 0xb0788cb3d228dafa expref: -99 pid: 21372 timeout: 0 lvb_type: 0 lru_score: 254 lru_type: 1 00000080:00200000:2.0:1787827175.966540:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000007:0x1:0x0](ffff8b2e98d1aa08), inode mode 41ff mask 1 00000080:00200000:2.0:1787827175.966543:0:21400:0:(dcache.c:276:ll_revalidate_dentry()) VFS Op:name=lockdir, flags=24 00000080:00200000:2.0:1787827175.966545:0:21400:0:(file.c:6689:ll_inode_permission()) VFS Op:inode=[0x200000401:0x20:0x0](ffff8b2e98de80c8), inode mode 41ed mask 1 00000080:00200000:2.0:1787827175.966551:0:21400:0:(namei.c:1335:ll_lookup_nd()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), flags=8 00000080:00200000:2.0:1787827175.966554:0:21400:0:(namei.c:1072:ll_lookup_it()) VFS Op:name=lockfile, dir=[0x200000401:0x20:0x0](ffff8b2e98de80c8), intent=getattr 00000002:00010000:2.0:1787827175.966559:0:21400:0:(mdc_locks.c:1508:mdc_intent_lock()) (name: lockfile,[0x0:0x0:0x0]) in obj [0x200000401:0x20:0x0], intent: getattr flags 01 00010000:00010000:2.0:1787827175.966575:0:21400:0:(ldlm_lock.c:775:ldlm_lock_addref_internal_nolock()) ### ldlm_lock_addref(CR) ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afc00/0x61d2feb329d0d262 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x0/0x0 rrc: 3 type: IBT flags: 0x10000000000000 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.966580:0:21400:0:(ldlm_request.c:1107:ldlm_cli_enqueue()) ### client-side enqueue START, flags 0x1000 ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afc00/0x61d2feb329d0d262 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00010000:00010000:2.0:1787827175.966585:0:21400:0:(ldlm_request.c:1194:ldlm_cli_enqueue()) ### sending request ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e869afc00/0x61d2feb329d0d262 lrc: 3/1,0 mode: --/CR res: [0x200000401:0x20:0x0].0x0 bits 0x2/0x0 rrc: 3 type: IBT flags: 0x0 pid: 21400 initiator: MDT0 lru_score: 0 lru_type: 0 00000100:00100000:2.0:1787827175.966593:0:21400:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803ed00 pname:cluuid:pid:xid:nid:opc:job createmany:31a8f82c-9690-4198-830f-493f54020135:21400:1874672517696128:192.168.203.101@tcp:101:createmany.0 00000100:00100000:2.0:1787827175.966624:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 11 seconds 00010000:00010000:1.0:1787827176.660805:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1787827177.364877:0:38:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x8:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827177.364893:0:38:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000008:00010000:2.0:1787827177.364924:0:38:0:(osc_cache.c:2020:try_to_add_extent_for_io()) ### extent: 000000007d20b40f ns: lustre-OST0000-osc-ffff8b2e901ad800 lock: 00000000e42cb945/0x61d2feb329d08eed lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d228cc59 expref: -99 pid: 15615 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00000008:00010000:2.0:1787827177.364944:0:38:0:(osc_cache.c:1174:osc_extent_make_ready()) ### extent: 000000007d20b40f ns: lustre-OST0000-osc-ffff8b2e901ad800 lock: 00000000e42cb945/0x61d2feb329d08eed lrc: 2/0,0 mode: PW/PW res: [0x240000400:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d228cc59 expref: -99 pid: 15615 timeout: 0 lvb_type: 1 lru_score: 2 lru_type: 0 00010000:00010000:2.0:1787827177.364998:0:38:0:(ldlm_lock.c:1522:ldlm_lock_match_with_skip()) ### matched (0 4095) ns: lustre-OST0000-osc-ffff8b2e901ad800 lock: 00000000e42cb945/0x61d2feb329d08eed lrc: 3/0,0 mode: PW/PW res: [0x240000400:0x5:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800020000000000 nid: local remote: 0xb0788cb3d228cc59 expref: -99 pid: 15615 timeout: 0 lvb_type: 1 lru_score: 3 lru_type: 0 00000008:00100000:2.0:1787827177.365084:0:38:0:(osc_request.c:2124:osc_brw_prep_request()) brw rpc ffff8b2e8803c780 - object 0x240000400:5 offset 0<>5 00000080:00200000:2.0:1787827177.365124:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x8:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1787827178.708830:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827179.924919:0:2350:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:31a8f82c-9690-4198-830f-493f54020135:2350:1874672517696512:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1787827179.925005:0:2350:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:31a8f82c-9690-4198-830f-493f54020135:2350:1874672517696768:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1787827179.925044:0:2350:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2350:1874672517697024:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1787827179.925077:0:2350:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:2.0:1787827179.925084:0:2350:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2350:1874672517697152:192.168.203.101@tcp:400:kworker.0 00010000:00010000:1.0:1787827180.756840:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1787827181.460793:0:38:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787827181.460807:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x9:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00010000:00010000:1.0:1787827181.780875:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827183.828831:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827185.044985:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fda940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:31a8f82c-9690-4198-830f-493f54020135:2348:1874672517697408:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827185.045074:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fd8000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:31a8f82c-9690-4198-830f-493f54020135:2348:1874672517697664:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827185.045118:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fd8b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517697920:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827185.045166:0:2348:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [0->1] 00000100:00100000:1.0:1787827185.045180:0:2348:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87fd9680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517698048:192.168.203.101@tcp:400:kworker.0 00010000:00010000:1.0:1787827185.876868:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827186.900809:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827187.092754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 1 seconds 00000100:00100000:2.0:1787827188.116751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 1 seconds 00010000:00010000:1.0:1787827188.948771:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827189.140730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 1 seconds 00000100:00100000:2.0:1787827190.164805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 1 seconds 00000100:00100000:1.0:1787827190.164975:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:31a8f82c-9690-4198-830f-493f54020135:2347:1874672517698304:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827190.165058:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:31a8f82c-9690-4198-830f-493f54020135:2347:1874672517698560:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827190.165089:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517698816:192.168.203.101@tcp:400:kworker.0 00010000:00010000:1.0:1787827190.996866:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00000400:2.0:1787827191.188826:0:21400:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827175/real 1787827175] req@ffff8b2e8803ed00 x1874672517696128/t0(0) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 584/1152 e 0 to 1 dl 1787827191 ref 2 fl Rpc:XQr/600/ffffffff rc 0/-1 job:'createmany.0' uid:0 gid:0 projid:0 00000100:00080000:1.0:1787827191.189060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827191.189065:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827191.189068:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827191.189076:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 119 00000100:00080000:1.0:1787827191.189094:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827191.189112:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827191.189130:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517699584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827191.189239:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58dbfc8-99f9-40e7-8102-506d8349451a:2347:1874672517699072:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827191.189337:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:31a8f82c-9690-4198-830f-493f54020135:2347:1874672517699328:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827191.189371:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517699712:192.168.203.101@tcp:400:kworker.0 00000100:00080000:2.0:1787827191.205184:0:21400:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1787827191.205195:0:21400:0:(import.c:182:ptlrpc_set_import_discon()) lustre-MDT0000-mdc-ffff8b2e901ad800: Connection to lustre-MDT0000 (at 192.168.203.101@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1787827191.205201:0:21400:0:(import.c:418:ptlrpc_pinger_force()) lustre-MDT0000_UUID: waking up pinger s:DISCONN 00000100:00100000:2.0:1787827191.205225:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827191.205391:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517699968:192.168.203.101@tcp:38:kworker.0 00010000:00010000:1.0:1787827192.020884:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827192.212789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827192.212841:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000080:00200000:1.0:1787827192.724785:0:38:0:(vvp_io.c:1889:vvp_io_init()) [0x200000402:0x2:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1787827192.724800:0:38:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787827192.724823:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000402:0x2:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:1.0:1787827192.724834:0:38:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:1.0:1787827192.724837:0:38:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:1.0:1787827192.724852:0:38:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1787827193.236801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827193.259270:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87eaf840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58dbfc8-99f9-40e7-8102-506d8349451a:2347:1874672517700096:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827193.259400:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87f6bc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517700480:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1787827193.259454:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87eae1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517700224:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827194.068972:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827194.260871:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827195.284776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00000400:2.0:1787827195.284806:0:2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827179/real 1787827179] req@ffff8b2e8803cf00 x1874672517697152/t0(0) o400->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827195.284848:0:2350:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e8803cf00 x1874672517697152/t0(0) o400->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827195 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827195.284860:0:2350:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1787827195.284865:0:2350:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff8b2e84abe800: Connection to lustre-OST0001 (at 192.168.203.101@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1787827195.299479:0:2350:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00000400:2.0:1787827195.299530:0:2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827179/real 1787827179] req@ffff8b2e8803d680 x1874672517696512/t0(0) o400->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827195.299552:0:2350:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e8803d680 x1874672517696512/t0(0) o400->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827195 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827195.299561:0:2350:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8b2e84b10800 already not connected (conn 2, was 1): CONNECTING 00000100:00000400:2.0:1787827195.299568:0:2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827179/real 1787827179] req@ffff8b2e8803c3c0 x1874672517696768/t0(0) o400->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827195.299579:0:2350:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e8803c3c0 x1874672517696768/t0(0) o400->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827195 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827195.299594:0:2350:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1787827195.299600:0:2350:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0001-osc-ffff8b2e901ad800: Connection to lustre-OST0001 (at 192.168.203.101@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1787827195.299603:0:2350:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0001_UUID: waking up pinger s:DISCONN 00000100:00000400:2.0:1787827195.299634:0:2350:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827179/real 1787827179] req@ffff8b2e8803c000 x1874672517697024/t0(0) o400->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827195 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827195.299668:0:2350:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e8803c000 x1874672517697024/t0(0) o400->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827195 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827195.299677:0:2350:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from FULL to DISCONN 00000100:02000400:2.0:1787827195.299682:0:2350:0:(import.c:182:ptlrpc_set_import_discon()) lustre-OST0000-osc-ffff8b2e84abe800: Connection to lustre-OST0000 (at 192.168.203.101@tcp) was lost; in progress operations using this service will wait for recovery to complete 00000100:00080000:2.0:1787827195.299685:0:2350:0:(import.c:418:ptlrpc_pinger_force()) lustre-OST0000_UUID: waking up pinger s:DISCONN 00000100:00100000:2.0:1787827195.299701:0:2350:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2350:1874672517697152:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1787827195.299709:0:2350:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:31a8f82c-9690-4198-830f-493f54020135:2350:1874672517696512:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1787827195.299715:0:2350:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:31a8f82c-9690-4198-830f-493f54020135:2350:1874672517696768:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1787827195.299719:0:2350:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_01_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2350:1874672517697024:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1787827195.299776:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff6580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517700864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827195.299861:0:2347:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e87ff5e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:b58dbfc8-99f9-40e7-8102-506d8349451a:2347:1874672517700736:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1787827195.299880:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517700992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827195.299917:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517701120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827195.306026:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87f652c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:b58dbfc8-99f9-40e7-8102-506d8349451a:2348:1874672517698176:192.168.203.101@tcp:400:kworker.0 00000100:00080000:2.0:1787827195.306028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827195.306042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827195.306048:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827195.306060:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 93 00000100:00080000:2.0:1787827195.306076:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827195.306104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803c000 x1874672517701248/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827195.306135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827195.306141:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827195.306145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827195.306149:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827195.306153:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827195.306157:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1787827196.119137:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827196.308784:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827196.308795:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827196.308799:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827196.308804:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827196.308808:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827196.308812:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827196.308819:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827196.308824:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827196.308828:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827196.308832:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827196.308834:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827196.308839:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827196.308846:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827196.308850:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:2.0:1787827196.308920:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827197.140835:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827197.332783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827197.332841:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827198.356780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827198.356806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827199.188841:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827199.380774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827200.212874:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827200.404751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787827200.404906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827200.404912:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827200.404915:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827200.404919:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827200.404922:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827200.404925:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827200.404927:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827200.404929:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827200.404930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827200.404932:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827200.404934:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827200.404936:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827200.404938:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827200.404940:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827200.405127:0:2350:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != CONNECTING) req@ffff8b2e8803ad00 x1874672517701376/t0(0) o63->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:24/4 lens 664/4320 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ll_sa.0' uid:0 gid:0 projid:4294967295 00010000:00010000:1.0:1787827201.236855:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827201.428796:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827191/real 1787827191] req@ffff8b2e8803b840 x1874672517699968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827201 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1787827201.428806:0:2347:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827185/real 1787827185] req@ffff8b2e87fdb0c0 x1874672517697536/t0(0) o400->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827201 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827201.428859:0:2347:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e87fdb0c0 x1874672517697536/t0(0) o400->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827201 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827201.428860:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517699968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827201 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827201.428882:0:2347:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8b2e88a45800 already not connected (conn 3, was 2): CONNECTING 00000100:00100000:2.0:1787827201.428888:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827191/real 1787827191] req@ffff8b2e880770c0 x1874672517699584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827201 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00000400:1.0:1787827201.428892:0:2347:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827185/real 1787827185] req@ffff8b2e87fd9a40 x1874672517697792/t0(0) o400->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827201 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827201.428920:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517699584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827201 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827201.428923:0:2347:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e87fd9a40 x1874672517697792/t0(0) o400->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827201 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827201.428947:0:2347:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8b2e87378000 already not connected (conn 2, was 1): CONNECTING 00000100:00100000:1.0:1787827201.428965:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fdb0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:31a8f82c-9690-4198-830f-493f54020135:2347:1874672517697536:192.168.203.101@tcp:400:kworker.0 00000100:00080000:2.0:1787827201.428967:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827201.428972:0:2347:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87fd9a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_00:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2347:1874672517697792:192.168.203.101@tcp:400:kworker.0 00000100:00080000:2.0:1787827201.428999:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827201.429006:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517699968:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827201.429015:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827201.429024:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827201.429028:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517699584:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827201.429203:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517701504:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827201.429330:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517701632:192.168.203.101@tcp:38:kworker.0 00010000:00010000:1.0:1787827202.260859:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827202.452909:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000080:00200000:2.0:1787827202.964786:0:36:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1d:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827202.964803:0:36:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827202.964830:0:36:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1d:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000080:00200000:2.0:1787827202.964845:0:36:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00020000:00200000:2.0:1787827202.964851:0:36:0:(lov_io.c:1253:lov_io_iter_init()) component[0] flags 0x10 00000080:00200000:2.0:1787827202.964859:0:36:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 0 00000100:00100000:2.0:1787827203.476781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827203.476823:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827193/real 1787827193] req@ffff8b2e87eae1c0 x1874672517700224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827203 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827203.476832:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827203.476873:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e87eae1c0 x1874672517700224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827203 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827203.476893:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827203.476918:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827203.476923:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87eae1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517700224:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787827203.476944:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827203.476949:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827203.476952:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827203.476955:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827203.476956:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827203.476958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827203.476961:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827203.476969:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 204 00000100:00080000:2.0:1787827203.476979:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 5s 00000100:00080000:2.0:1787827203.476985:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827203.476995:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827203.477025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803c3c0 x1874672517701760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827203.477045:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827203.477048:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827203.477050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827203.477052:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827203.477053:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827203.477055:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827203.477057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827203.477059:0:36:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827203.477098:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517701760:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827204.308837:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827204.500749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827205.332908:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827205.524780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827205.524886:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827206.356812:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827206.548805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00000400:1.0:1787827206.548867:0:2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827190/real 1787827190] req@ffff8b2e880752c0 x1874672517698432/t0(0) o400->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827206.548913:0:2348:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e880752c0 x1874672517698432/t0(0) o400->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827206 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827206.548925:0:2348:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8b2e88a45800 already not connected (conn 4, was 2): CONNECTING 00000100:00000400:1.0:1787827206.548935:0:2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827190/real 1787827190] req@ffff8b2e88077840 x1874672517698688/t0(0) o400->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827206.548950:0:2348:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e88077840 x1874672517698688/t0(0) o400->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827206 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827206.548958:0:2348:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8b2e87378000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:1.0:1787827206.548965:0:2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827190/real 1787827190] req@ffff8b2e88074780 x1874672517698944/t0(0) o400->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827206 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827206.548979:0:2348:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e88074780 x1874672517698944/t0(0) o400->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827206 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827206.548987:0:2348:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8b2e90dee000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:1.0:1787827206.549001:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:31a8f82c-9690-4198-830f-493f54020135:2348:1874672517698432:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827206.549008:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517698688:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827206.549014:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517698944:192.168.203.101@tcp:400:kworker.0 00010000:00010000:1.0:1787827207.380877:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827207.572742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00000400:1.0:1787827207.572840:0:2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827191/real 1787827191] req@ffff8b2e88074f00 x1874672517699840/t0(0) o400->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827207.572903:0:2348:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e88074f00 x1874672517699840/t0(0) o400->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827207 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827207.572912:0:2348:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8b2e90dee000 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:1.0:1787827207.572922:0:2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827191/real 1787827191] req@ffff8b2e88075680 x1874672517699200/t0(0) o400->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827207.572944:0:2348:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e88075680 x1874672517699200/t0(0) o400->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 224/224 e 0 to 1 dl 1787827207 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827207.572951:0:2348:0:(import.c:214:ptlrpc_set_import_discon()) mdc: import ffff8b2e84b10800 already not connected (conn 3, was 1): CONNECTING 00000100:00000400:1.0:1787827207.572960:0:2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827191/real 1787827191] req@ffff8b2e880743c0 x1874672517699456/t0(0) o400->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827207 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827207.572975:0:2348:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e880743c0 x1874672517699456/t0(0) o400->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827207 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827207.572983:0:2348:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8b2e88a47800 already not connected (conn 4, was 2): CONNECTING 00000100:00100000:1.0:1787827207.573000:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517699840:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827207.573007:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:31a8f82c-9690-4198-830f-493f54020135:2348:1874672517699200:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827207.573012:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:31a8f82c-9690-4198-830f-493f54020135:2348:1874672517699456:192.168.203.101@tcp:400:kworker.0 00000100:00100000:2.0:1787827208.596734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827208.596776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827209.428823:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827209.620746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00000400:1.0:1787827209.620789:0:2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827193/real 1787827193] req@ffff8b2e8802a580 x1874672517700352/t0(0) o400->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827209.620825:0:2348:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e8802a580 x1874672517700352/t0(0) o400->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827209 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827209.620840:0:2348:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8b2e88a47800 already not connected (conn 4, was 2): CONNECTING 00000100:00000400:1.0:1787827209.620847:0:2348:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827193/real 1787827193] req@ffff8b2e87eca940 x1874672517700608/t0(0) o400->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827209 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827209.620869:0:2348:0:(client.c:2547:ptlrpc_expire_one_request()) @@@ TIMEOUT-NORESEND: req@ffff8b2e87eca940 x1874672517700608/t0(0) o400->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 224/224 e 0 to 1 dl 1787827209 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827209.620875:0:2348:0:(import.c:214:ptlrpc_set_import_discon()) osc: import ffff8b2e90dee000 already not connected (conn 3, was 1): CONNECTING 00000100:00100000:1.0:1787827209.620882:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8802a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:31a8f82c-9690-4198-830f-493f54020135:2348:1874672517700352:192.168.203.101@tcp:400:kworker.0 00000100:00100000:1.0:1787827209.620887:0:2348:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e87eca940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_00_01:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2348:1874672517700608:192.168.203.101@tcp:400:kworker.0 00010000:00010000:1.0:1787827210.452814:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827210.580823:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827210.580836:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827210.580840:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827210.580844:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827210.580847:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827210.580850:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827210.580852:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827210.580855:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827210.580857:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827210.580860:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827210.580862:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827210.580864:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827210.580865:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827210.580867:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1787827211.476840:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827212.500849:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827214.548780:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827214.740862:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827215.572802:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827215.764736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827216.596798:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827216.788765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827216.788778:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827201/real 1787827201] req@ffff8b2e88038b40 x1874672517701632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827216 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827216.788804:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038b40 x1874672517701632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827216 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827216.788815:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827201/real 1787827201] req@ffff8b2e88038780 x1874672517701504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827216 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827216.788831:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038780 x1874672517701504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827216 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827216.788853:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827216.788874:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827216.788879:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517701632:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827216.788885:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827216.788886:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827216.788892:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787827216.788894:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827216.788899:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517701504:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827216.788901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827216.788908:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827216.788912:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827216.788920:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 212 00000100:00080000:2.0:1787827216.788933:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1787827216.788942:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827216.788953:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827216.788987:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803d680 x1874672517702400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827216.789005:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827216.789008:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827216.789010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827216.789012:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827216.789014:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827216.789016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827216.789018:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827216.789022:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 212 00000100:00080000:2.0:1787827216.789026:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1787827216.789029:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827216.789032:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827216.789035:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517702400:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827216.789043:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803cf00 x1874672517702528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827216.789052:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827216.789054:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827216.789056:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827216.789058:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827216.789064:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827216.789066:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827216.789068:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827216.789071:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827216.789072:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827216.789075:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827216.789076:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827216.789079:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827216.789081:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827216.789082:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827216.789135:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517702528:192.168.203.101@tcp:38:kworker.0 00010000:00010000:1.0:1787827217.620833:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827217.812764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827218.644793:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827218.836776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827218.836787:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827203/real 1787827203] req@ffff8b2e8803c3c0 x1874672517701760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827218 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827218.836821:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803c3c0 x1874672517701760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827218 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827218.836848:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827218.836864:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827218.836868:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517701760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827218.837052:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517702656:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827219.668814:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827219.860783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827220.692832:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827220.884772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827220.884841:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8b2e88040780 x1874672517702272/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.884875:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517702272/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.884884:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8b2e88043840 x1874672517702016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.884899:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517702016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.884910:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8b2e88042d00 x1874672517702144/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.884922:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517702144/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.884933:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827205/real 1787827205] req@ffff8b2e880421c0 x1874672517701888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827220.884946:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517701888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827220 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827220.884970:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827220.885002:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827220.885009:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517702272:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827220.885016:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827220.885023:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827220.885027:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517702016:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827220.885031:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827220.885056:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827220.885062:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517702144:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787827220.885070:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827220.885082:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827220.885086:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827220.885087:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827220.885097:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827220.885102:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827220.885106:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827220.885109:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827220.885113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827220.885117:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827220.885119:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827220.885131:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517701888:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787827220.885149:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 217 00000100:00080000:2.0:1787827220.885163:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1787827220.885173:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827220.885189:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827220.885233:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803c3c0 x1874672517702784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827220.885269:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827220.885273:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827220.885276:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827220.885280:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827220.885282:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827220.885287:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 217 00000100:00080000:2.0:1787827220.885291:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1787827220.885296:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827220.885302:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827220.885314:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803c000 x1874672517702912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827220.885322:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827220.885324:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827220.885325:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827220.885327:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 217 00000100:00080000:2.0:1787827220.885330:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1787827220.885332:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827220.885337:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827220.885348:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803f0c0 x1874672517703040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827220.885363:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827220.885367:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827220.885369:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827220.885373:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 217 00000100:00080000:2.0:1787827220.885376:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 10s 00000100:00080000:2.0:1787827220.885380:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827220.885395:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803de00 x1874672517703168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827220.885409:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827220.885412:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827220.885420:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517702784:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787827220.885420:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827220.885457:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827220.885461:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827220.885464:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827220.885609:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517702912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827220.885676:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517703040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827220.885716:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517703168:192.168.203.101@tcp:250:kworker.0 00010000:00010000:1.0:1787827221.716851:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827221.908772:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827221.908780:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827221.908783:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827221.908787:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827221.908790:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827221.908793:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827221.908796:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827221.908800:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827221.908804:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827221.908807:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827221.908810:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827221.908814:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827221.908826:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827221.908830:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1787827222.740946:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827223.764806:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827223.956727:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.956735:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827223.956738:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.956741:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827223.956743:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.956745:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827223.956747:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.956749:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827223.956751:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.956760:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827223.956766:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.956769:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827223.956772:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827223.956775:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1787827224.788839:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827225.812829:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827226.004739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787827226.004841:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827226.004846:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827226.004849:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827226.004852:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827226.004854:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827226.004856:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827226.004858:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827226.004861:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827226.004864:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827226.004866:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827226.004869:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827226.004871:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827226.004873:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827226.004875:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1787827226.836825:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827227.860806:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827228.884837:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827229.076914:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827229.908867:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827230.100793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827230.932842:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827231.124820:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827231.956808:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827232.148920:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827232.980839:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827233.172767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827234.004847:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827234.196779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827235.028877:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827236.052861:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827236.244775:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827216/real 1787827216] req@ffff8b2e8803cf00 x1874672517702528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827236.244810:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803cf00 x1874672517702528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827236 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827236.244823:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827216/real 1787827216] req@ffff8b2e8803d680 x1874672517702400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827236 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827236.244835:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803d680 x1874672517702400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827236 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827236.244852:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827236.244869:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827236.244873:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517702528:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827236.244877:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827236.244881:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827236.244884:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517702400:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827236.244902:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827236.244909:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827236.244913:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827236.244916:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827236.244920:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827236.244927:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 228 00000100:00080000:2.0:1787827236.244938:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787827236.244945:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827236.244958:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827236.244990:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803cb40 x1874672517703424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827236.245018:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827236.245022:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827236.245026:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827236.245030:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827236.245033:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827236.245038:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827236.245042:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827236.245047:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 228 00000100:00080000:2.0:1787827236.245052:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787827236.245056:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827236.245062:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827236.245075:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803f840 x1874672517703552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827236.245086:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827236.245090:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827236.245093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827236.245097:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827236.245133:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517703424:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827236.245274:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517703552:192.168.203.101@tcp:38:kworker.0 00010000:00010000:1.0:1787827237.076818:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827238.100838:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000800:00000100:2.0:1787827238.164822:0:2338:0:(socklnd_cb.c:2254:ksocknal_find_timed_out_conn()) A connection with 12345-192.168.203.101@tcp (192.168.203.101:988) timed out; the network or node may be down. 00000100:00100000:1.0:1787827238.292821:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827218/real 1787827218] req@ffff8b2e88040b40 x1874672517702656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827238 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827238.292853:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517702656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827238 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827238.292872:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827238.292887:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827238.292891:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517702656:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787827238.292932:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827238.292940:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827238.292946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827238.292950:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827238.292952:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827238.292955:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827238.292969:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827238.292973:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827238.292982:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 230 00000100:00080000:2.0:1787827238.292996:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787827238.293005:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827238.293019:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827238.293063:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803d2c0 x1874672517703680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827238.293094:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827238.293098:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827238.293101:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827238.293105:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827238.293116:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827238.293120:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827238.293124:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827238.293128:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00100000:1.0:1787827238.293136:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517703680:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827239.124894:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827239.316779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827240.148826:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827240.340782:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827220/real 1787827220] req@ffff8b2e8803de00 x1874672517703168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827240.340821:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803de00 x1874672517703168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827240.340834:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827220/real 1787827220] req@ffff8b2e8803c3c0 x1874672517702784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827240.340849:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803c3c0 x1874672517702784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827240.340862:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827220/real 1787827220] req@ffff8b2e8803c000 x1874672517702912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827240.340879:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803c000 x1874672517702912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827240.340895:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827220/real 1787827220] req@ffff8b2e8803f0c0 x1874672517703040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827240.340909:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803f0c0 x1874672517703040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827240 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827240.340930:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827240.341010:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827240.341016:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517703168:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827240.341024:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827240.341091:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341098:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827240.341101:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827240.341109:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 232 00000100:00080000:2.0:1787827240.341121:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787827240.341130:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827240.341167:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803e940 x1874672517703808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827240.341186:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341191:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827240.341195:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341199:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827240.341201:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341205:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827240.341207:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827240.341210:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 232 00000100:00080000:2.0:1787827240.341214:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787827240.341218:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827240.341229:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827240.341242:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803d680 x1874672517703936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827240.341252:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341256:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827240.341259:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341262:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827240.341265:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341269:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827240.341369:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827240.341375:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517702784:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827240.341383:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827240.341406:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341410:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827240.341412:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341415:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827240.341419:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341422:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827240.341424:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341427:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827240.341429:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827240.341433:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 232 00000100:00080000:2.0:1787827240.341437:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787827240.341441:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827240.341447:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827240.341461:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803cf00 x1874672517704064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827240.341473:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341476:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827240.341556:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827240.341561:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517702912:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827240.341567:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827240.341590:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341594:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827240.341597:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341600:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827240.341603:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341606:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827240.341609:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 1 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.341612:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827240.341614:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827240.341618:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 232 00000100:00080000:2.0:1787827240.341622:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 15s 00000100:00080000:2.0:1787827240.341626:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827240.341632:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827240.341668:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88009e00 x1874672517704192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827240.341774:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827240.341779:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517703040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827240.341814:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517703808:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827240.341901:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517703936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827240.341935:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517704064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827240.341957:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88009e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517704192:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787827240.342020:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.342024:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827240.342027:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.342030:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827240.342032:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827240.342035:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00010000:00010000:1.0:1787827241.172832:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827241.364794:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827241.364805:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827241.364810:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827241.364814:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827241.364817:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827241.364819:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827241.364821:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827241.364824:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827241.364827:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827241.364830:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827241.364832:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827241.364834:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827241.364837:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827241.364839:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000400:00000100:2.0:1787827241.365036:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.203.101:988 00000400:00000100:2.0:1787827241.365078:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827241.365096:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827241.365129:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827241.365135:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827241.365142:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827241.365148:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:1.0:1787827241.365169:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827240/real 1787827241] req@ffff8b2e8803e940 x1874672517703808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827241.365192:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803e940 x1874672517703808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827241.365203:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827241.365209:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827241.365214:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517703808:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827241.365222:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827240/real 1787827241] req@ffff8b2e8803d680 x1874672517703936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827241.365242:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803d680 x1874672517703936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827241.365270:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827241.365275:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827241.365279:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517703936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827241.365286:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827240/real 1787827241] req@ffff8b2e8803cf00 x1874672517704064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827241.365316:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803cf00 x1874672517704064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827241.365326:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827241.365330:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827241.365333:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517704064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827241.365340:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827240/real 1787827241] req@ffff8b2e88009e00 x1874672517704192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827265 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827241.365351:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88009e00 x1874672517704192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827265 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827241.365359:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827241.365363:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827241.365366:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88009e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517704192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827241.365372:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827238/real 1787827241] req@ffff8b2e8803d2c0 x1874672517703680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827241.365383:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803d2c0 x1874672517703680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827241.365391:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827241.365394:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827241.365398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517703680:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827242.196860:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827243.220810:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827243.412747:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827243.412759:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827243.412764:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827243.412775:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 251 00000100:00080000:2.0:1787827243.412787:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827243.412794:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827243.412838:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88009e00 x1874672517704320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827243.412865:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827243.412870:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827243.412875:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827243.412879:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827243.412882:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827243.412888:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 249 00000100:00080000:2.0:1787827243.412893:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827243.412900:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827243.412901:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88009e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517704320:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787827243.412915:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827243.412954:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517704448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827243.412964:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827243.412967:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827243.412968:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827243.412971:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 251 00000100:00080000:2.0:1787827243.412975:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827243.412979:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827243.412982:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827243.412990:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517704576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827243.412997:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827243.413001:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827243.413003:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827243.413005:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827243.413008:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827243.413012:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 251 00000100:00080000:2.0:1787827243.413015:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827243.413020:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517704448:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787827243.413021:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827243.413030:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827243.413046:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517704704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827243.413062:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827243.413066:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517704576:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787827243.413069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827243.413074:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827243.413080:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 251 00000100:00080000:2.0:1787827243.413086:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827243.413090:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517704704:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787827243.413094:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827243.413102:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827243.413117:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517704832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827243.413145:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517704832:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827244.244853:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827244.436838:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827244.437260:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827243/real 1787827244] req@ffff8b2e8801da40 x1874672517704704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827244.437316:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672517704704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827244.437332:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827244.437338:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827244.437344:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517704704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827244.437359:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827243/real 1787827244] req@ffff8b2e8801c000 x1874672517704448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827244.437376:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672517704448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827244.437386:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827244.437391:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827244.437395:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517704448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827244.437404:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827243/real 1787827244] req@ffff8b2e8801cf00 x1874672517704576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827244.437418:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672517704576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827244.437427:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827244.437431:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827244.437435:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517704576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827244.437443:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827243/real 1787827244] req@ffff8b2e88009e00 x1874672517704320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827244.437458:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88009e00 x1874672517704320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827244.437467:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827244.437470:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827244.437474:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88009e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517704320:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827244.437513:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827243/real 1787827244] req@ffff8b2e8801fc00 x1874672517704832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827244.437527:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517704832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827244.437542:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827244.437546:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827244.437553:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517704832:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827245.268825:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827245.460783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827246.292905:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827246.484741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827247.316866:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827248.340816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827248.532745:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827248.532754:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827248.532758:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827248.532769:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 254 00000100:00080000:2.0:1787827248.532781:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827248.532790:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827248.532830:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517704960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827248.532859:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827248.532863:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827248.532866:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827248.532869:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827248.532872:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827248.532876:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 254 00000100:00080000:2.0:1787827248.532880:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827248.532884:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827248.532894:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827248.532907:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517705088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827248.532917:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827248.532920:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827248.532922:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827248.532926:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 254 00000100:00080000:2.0:1787827248.532929:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827248.532933:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827248.532937:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827248.532947:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517705216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827248.532957:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827248.532960:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:2.0:1787827248.532963:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827248.532966:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827248.532969:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827248.532973:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 254 00000100:00080000:2.0:1787827248.532976:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827248.532981:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827248.532987:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827248.533000:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d2c0 x1874672517705344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827248.533009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827248.533014:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827248.533016:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827248.533020:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 254 00000100:00080000:2.0:1787827248.533023:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827248.533027:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827248.533032:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827248.533044:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517704960:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827248.533047:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517705472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827248.533207:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517705088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827248.533259:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517705216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827248.533301:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517705344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827248.533340:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517705472:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827249.364830:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827250.388827:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827250.581020:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.203.101:988 00000400:00000100:2.0:1787827250.581058:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827250.581074:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827250.581108:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827250.581118:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827250.581128:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827250.581139:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:1.0:1787827250.581145:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827248/real 1787827250] req@ffff8b2e8801f480 x1874672517704960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827250.581167:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517704960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827250.581179:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827250.581184:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827250.581188:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517704960:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827250.581195:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827248/real 1787827250] req@ffff8b2e8801e940 x1874672517705088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827250.581209:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517705088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827250.581216:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827250.581220:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827250.581239:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517705088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827250.581247:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827248/real 1787827250] req@ffff8b2e8801e580 x1874672517705216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827250.581260:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517705216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827250.581267:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827250.581271:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827250.581274:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517705216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827250.581281:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827248/real 1787827250] req@ffff8b2e8801d2c0 x1874672517705344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827250.581293:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d2c0 x1874672517705344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827250.581301:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827250.581304:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827250.581307:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517705344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827250.581332:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827248/real 1787827250] req@ffff8b2e8801e1c0 x1874672517705472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827250.581343:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e1c0 x1874672517705472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827250.581351:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827250.581354:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827250.581357:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517705472:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827251.412799:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827252.436845:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827252.628744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827253.460827:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827253.652746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827253.652777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827253.653064:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517705600:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787827253.653155:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827253.653232:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827253/real 1787827253] req@ffff8b2e88041e00 x1874672517705600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827253.653250:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517705600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827253.653259:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827253.653262:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827253.653266:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517705600:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827253.653284:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517705728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827253.653306:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827253.653315:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827253/real 1787827253] req@ffff8b2e880403c0 x1874672517705728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827253.653325:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672517705728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827253.653333:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827253.653335:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827253.653338:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517705728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827253.653343:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517705856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827253.653359:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827253.653367:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827253/real 1787827253] req@ffff8b2e880430c0 x1874672517705856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827253.653379:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517705856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827253.653387:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827253.653390:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827253.653393:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517705856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827253.653400:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517705984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827253.653422:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827253.653435:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827253/real 1787827253] req@ffff8b2e88040000 x1874672517705984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827253.653448:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517705984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827253.653457:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827253.653460:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827253.653464:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517705984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827253.653471:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517706112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827253.653491:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827253.653500:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827253/real 1787827253] req@ffff8b2e88040f00 x1874672517706112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827253.653508:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517706112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827253.653513:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827253.653515:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827253.653517:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517706112:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827254.484829:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827254.676740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827255.508894:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827255.700780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827256.532795:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827256.724736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827257.556842:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827257.748760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827258.580817:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827258.772730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787827258.772754:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.772760:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827258.772763:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827258.772773:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 265 00000100:00080000:1.0:1787827258.772781:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827258.772787:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827258.772818:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517706240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827258.772841:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.772848:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827258.772850:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.772854:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827258.772855:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827258.772858:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 265 00000100:00080000:1.0:1787827258.772861:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827258.772864:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827258.772873:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827258.772873:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517706240:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827258.772886:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517706368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827258.772894:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.772896:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827258.772897:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827258.772900:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 265 00000100:00080000:1.0:1787827258.772902:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827258.772904:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827258.772907:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827258.772917:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517706496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827258.772923:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level CONNECTING/4 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.772925:0:38:0:(pinger.c:247:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: not pinging (in recovery or recovery disabled: CONNECTING) 00000100:00080000:1.0:1787827258.772927:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.772929:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827258.772930:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827258.772933:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 265 00000100:00080000:1.0:1787827258.772935:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827258.772938:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827258.772940:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827258.772948:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517706624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827258.772953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827258.772955:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827258.772956:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827258.772959:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 265 00000100:00080000:1.0:1787827258.772961:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827258.772963:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827258.772965:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827258.772974:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517706752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827258.772992:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517706368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827258.773035:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517706496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827258.773060:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517706624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827258.773107:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517706752:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827259.604824:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827259.796777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827259.797109:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827258/real 1787827259] req@ffff8b2e88077840 x1874672517706752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827259.797135:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517706752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827259.797146:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827259.797151:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827259.797155:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517706752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827259.797175:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827258/real 1787827259] req@ffff8b2e88075680 x1874672517706368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827259.797190:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517706368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827259.797199:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827259.797202:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827259.797206:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517706368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827259.797213:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827258/real 1787827259] req@ffff8b2e88074f00 x1874672517706496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827259.797225:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517706496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827259.797232:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827259.797236:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827259.797239:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517706496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827259.797245:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827258/real 1787827259] req@ffff8b2e88074780 x1874672517706624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827259.797256:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517706624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827259.797264:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827259.797268:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827259.797271:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517706624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827259.797277:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827258/real 1787827259] req@ffff8b2e880743c0 x1874672517706240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827259.797290:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517706240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827259.797305:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827259.797309:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827259.797313:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517706240:192.168.203.101@tcp:250:kworker.0 00010000:00010000:1.0:1787827260.628825:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827260.820778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827261.652816:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827261.844777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827261.844783:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827236/real 1787827236] req@ffff8b2e8803cb40 x1874672517703424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827261.844812:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803cb40 x1874672517703424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827261.844826:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has timed out for slow reply: [sent 1787827236/real 1787827236] req@ffff8b2e8803f840 x1874672517703552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:XNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827261.844840:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803f840 x1874672517703552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827261 ref 1 fl Rpc:XNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827261.844855:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827261.844863:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827261.844867:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517703424:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827261.844872:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827261.844878:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827261.844881:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517703552:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827261.845072:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517706880:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827261.845186:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827261.845242:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff8b2e88041a40 x1874672517706880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827261.845275:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517706880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827261.845307:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827261.845317:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827261.845334:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517706880:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827261.845366:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517707008:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827261.845445:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827261.845477:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff8b2e88043c00 x1874672517707008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827261.845518:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043c00 x1874672517707008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827261.845547:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827261.845556:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827261.845566:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517707008:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827261.845592:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517707136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827261.845629:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827261.845643:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff8b2e88043480 x1874672517707136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827261.845692:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517707136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827261.845701:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827261.845705:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827261.845708:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517707136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827261.845715:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517707264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827261.845744:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827261.845758:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff8b2e88042940 x1874672517707264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827261.845769:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517707264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827261.845777:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827261.845780:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827261.845783:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517707264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827261.845800:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517707392:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827261.845822:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827261.845835:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff8b2e88040f00 x1874672517707392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827261.845848:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517707392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827261.845863:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827261.845867:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827261.845870:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517707392:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827261.845877:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517707520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827261.845900:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827261.845913:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff8b2e88040000 x1874672517707520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827261.845924:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517707520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827261.845933:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827261.845937:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827261.845941:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517707520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827261.845949:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517707648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827261.845973:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827261.845986:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827261/real 1787827261] req@ffff8b2e880430c0 x1874672517707648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827261.845998:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517707648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827286 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827261.846005:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827261.846008:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827261.846012:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517707648:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827262.676820:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827262.868737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827263.700821:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827263.892760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827264.724773:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827264.916735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827265.748821:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827265.940772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827266.772817:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827266.964731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827266.964979:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517707776:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827266.965052:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827266.965106:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827266/real 1787827266] req@ffff8b2e88039680 x1874672517707776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827266.965150:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517707776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827266.965169:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827266.965176:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827266.965183:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517707776:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827266.965217:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517707904:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827266.965255:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827266.965269:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827266/real 1787827266] req@ffff8b2e88039e00 x1874672517707904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827266.965284:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517707904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827266.965293:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827266.965296:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827266.965300:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517707904:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827266.965306:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517708032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827266.965327:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827266.965338:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827266/real 1787827266] req@ffff8b2e880383c0 x1874672517708032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827266.965349:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517708032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827266.965356:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827266.965359:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827266.965363:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517708032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827266.965368:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517708160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827266.965384:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827266.965395:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827266/real 1787827266] req@ffff8b2e8803b0c0 x1874672517708160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827266.965407:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517708160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827266.965414:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827266.965417:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827266.965420:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517708160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827266.965426:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517708288:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827266.965461:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827266.965474:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827266/real 1787827266] req@ffff8b2e88038780 x1874672517708288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827266.965486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038780 x1874672517708288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827266.965495:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827266.965499:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827266.965502:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517708288:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827266.965508:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517708416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827266.965529:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827266.965540:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827266/real 1787827266] req@ffff8b2e88038b40 x1874672517708416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827266.965552:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038b40 x1874672517708416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827266.965559:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827266.965562:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827266.965566:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517708416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827266.965573:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517708544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827266.965593:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827266.965604:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827266/real 1787827266] req@ffff8b2e8803b840 x1874672517708544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827266.965614:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517708544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827291 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827266.965621:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827266.965624:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827266.965628:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517708544:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827267.796828:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827267.988748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827268.820845:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827269.012766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827269.844891:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827270.036775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827270.036844:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827270.868819:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827271.060769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827271.060805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827271.892934:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827272.084737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827272.084740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827272.084858:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517708672:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827272.084963:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517708800:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827272.085000:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517708928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827272.085038:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517709056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827272.085095:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517709184:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827272.085186:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517709312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827272.085228:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517709440:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827272.916782:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827273.108751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827273.108752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827273.940807:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827274.132722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827274.132741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827274.964810:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827275.156713:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827275.156736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:1.0:1787827275.156926:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1021 -> 192.168.203.101:988 00000400:00000100:1.0:1787827275.156960:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827275.156975:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827275.157004:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827275.157011:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827275.157018:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827275.157025:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827275.157032:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:2.0:1787827275.157037:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827272/real 1787827275] req@ffff8b2e8803b480 x1874672517709056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787827275.157042:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:2.0:1787827275.157081:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517709056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827275.157093:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827275.157099:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827275.157103:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517709056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827275.157111:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827272/real 1787827275] req@ffff8b2e8803a1c0 x1874672517708800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827275.157124:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517708800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827275.157132:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827275.157136:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827275.157140:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517708800:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827275.157147:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827272/real 1787827275] req@ffff8b2e8803a580 x1874672517708928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827275.157158:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517708928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827275.157166:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827275.157169:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827275.157172:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517708928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827275.157179:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827272/real 1787827275] req@ffff8b2e880392c0 x1874672517708672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827275.157191:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517708672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827275.157199:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827275.157202:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827275.157205:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517708672:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827275.157235:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827272/real 1787827275] req@ffff8b2e8803b840 x1874672517709440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827275.157248:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517709440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827275.157255:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827275.157259:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827275.157263:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517709440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827275.157269:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827272/real 1787827275] req@ffff8b2e88038f00 x1874672517709312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827275.157279:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517709312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827275.157292:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827275.157295:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827275.157298:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517709312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827275.157304:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827272/real 1787827275] req@ffff8b2e88039a40 x1874672517709184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827275.157317:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517709184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827275.157325:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827275.157329:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827275.157332:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517709184:192.168.203.101@tcp:38:kworker.0 00010000:00010000:1.0:1787827275.988845:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827276.180706:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827276.180732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827277.012825:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827277.204777:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827277.204783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787827277.204792:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827277.204799:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827277.204812:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 283 00000100:00080000:2.0:1787827277.204827:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827277.204838:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827277.204882:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517709568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827277.204909:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827277.204914:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827277.204917:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827277.204923:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 283 00000100:00080000:2.0:1787827277.204928:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827277.204933:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827277.204944:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827277.204948:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517709568:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827277.204958:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517709696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827277.204969:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827277.204972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827277.204974:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827277.204978:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 283 00000100:00080000:2.0:1787827277.204981:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827277.204984:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827277.204989:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827277.204999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517709824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827277.205008:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827277.205011:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827277.205012:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827277.205015:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 283 00000100:00080000:2.0:1787827277.205018:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827277.205022:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:1.0:1787827277.205024:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787827277.205026:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827277.205035:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517709952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827277.205057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827277.205060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827277.205062:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827277.205067:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 283 00000100:00080000:2.0:1787827277.205070:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827277.205074:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827277.205079:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827277.205084:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827277/real 1787827277] req@ffff8b2e8801f840 x1874672517709568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827277.205089:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517710080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827277.205099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827277.205102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827277.205104:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827277.205105:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f840 x1874672517709568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827277.205109:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 283 00000100:00080000:2.0:1787827277.205113:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827277.205116:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:1.0:1787827277.205117:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787827277.205121:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827277.205122:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827277.205126:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517709568:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827277.205130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517710208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827277.205154:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827277.205156:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517709696:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827277.205158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827277.205160:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827277.205164:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 283 00000100:00080000:2.0:1787827277.205168:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827277.205172:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827277.205178:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827277.205186:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827277.205188:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517710336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827277.205201:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827277/real 1787827277] req@ffff8b2e8801ed00 x1874672517709696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827277.205215:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517709696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827277.205225:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827277.205229:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827277.205232:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517709696:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827277.205239:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517709824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827277.205267:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827277.205280:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827277/real 1787827277] req@ffff8b2e8801c780 x1874672517709824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827277.205292:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517709824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827277.205302:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827277.205306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827277.205309:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517709824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827277.205316:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517709952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827277.205340:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827277.205353:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827277/real 1787827277] req@ffff8b2e8801cb40 x1874672517709952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827277.205373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517709952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827277.205383:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827277.205386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827277.205390:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517709952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827277.205397:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517710080:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827277.205421:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827277.205434:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827277/real 1787827277] req@ffff8b2e8801d680 x1874672517710080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827277.205449:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517710080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827277.205458:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827277.205462:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827277.205466:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517710080:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827277.205472:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517710208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827277.205494:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827277.205506:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827277/real 1787827277] req@ffff8b2e8801de00 x1874672517710208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827277.205518:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517710208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827277.205534:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827277.205537:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827277.205540:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517710208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827277.205561:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517710336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827277.205582:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827277.205593:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827277/real 1787827277] req@ffff8b2e8801c3c0 x1874672517710336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827277.205606:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517710336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827277.205615:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827277.205618:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827277.205622:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517710336:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827278.036947:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827278.228744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827279.060809:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827279.252761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827280.084785:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827280.276803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827280.276808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827281.108840:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827281.300748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827281.300751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827282.132842:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827282.324761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827282.324766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827282.324956:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517710464:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787827282.325087:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827282.325165:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff8b2e88038b40 x1874672517710464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827282.325198:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038b40 x1874672517710464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827282.325224:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827282.325232:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827282.325241:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517710464:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827282.325288:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517710592:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827282.325319:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827282.325335:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff8b2e88038780 x1874672517710592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827282.325350:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038780 x1874672517710592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827282.325360:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827282.325364:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827282.325368:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517710592:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827282.325376:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517710720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827282.325402:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827282.325416:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff8b2e8803b0c0 x1874672517710720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827282.325437:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517710720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827282.325447:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827282.325451:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827282.325455:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517710720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827282.325462:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517710848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827282.325486:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827282.325498:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff8b2e880383c0 x1874672517710848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827282.325512:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517710848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827282.325522:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827282.325526:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827282.325530:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517710848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827282.325537:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517710976:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827282.325561:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827282.325574:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff8b2e88039e00 x1874672517710976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827282.325587:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517710976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827282.325603:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827282.325607:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827282.325611:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517710976:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827282.325617:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517711104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827282.325639:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827282.325682:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff8b2e88039680 x1874672517711104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827282.325695:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517711104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827282.325704:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827282.325708:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827282.325711:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517711104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827282.325718:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517711232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827282.325738:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827282.325750:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827282/real 1787827282] req@ffff8b2e88039a40 x1874672517711232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827282.325763:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517711232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827282.325772:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827282.325775:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827282.325779:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517711232:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827283.156897:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827283.348765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827283.348774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827284.180840:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827284.372747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827284.372779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827285.204939:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827285.396761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827285.396761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827286.228895:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827286.420730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827286.420772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827287.252839:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827287.444734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827287.444734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827287.444879:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517711360:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787827287.444953:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827287.445042:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827287/real 1787827287] req@ffff8b2e88038f00 x1874672517711360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827287.445086:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517711360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827287.445133:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827287.445163:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827287.445179:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517711360:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827287.445237:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517711488:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827287.445274:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827287.445298:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827287/real 1787827287] req@ffff8b2e8803b840 x1874672517711488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827287.445311:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517711488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827287.445321:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827287.445325:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827287.445329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517711488:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827287.445335:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517711616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827287.445358:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827287.445369:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827287/real 1787827287] req@ffff8b2e880392c0 x1874672517711616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827287.445381:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517711616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827287.445390:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827287.445394:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827287.445398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517711616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827287.445404:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517711744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827287.445432:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827287.445444:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827287/real 1787827287] req@ffff8b2e8803a580 x1874672517711744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827287.445456:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517711744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827287.445463:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827287.445466:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827287.445470:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517711744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827287.445477:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517711872:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827287.445496:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827287.445508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827287/real 1787827287] req@ffff8b2e8803a1c0 x1874672517711872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827287.445519:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517711872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827287.445528:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827287.445531:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827287.445534:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517711872:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827287.445540:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517712000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827287.445560:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827287.445571:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827287/real 1787827287] req@ffff8b2e8803b480 x1874672517712000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827287.445585:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517712000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827287.445592:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827287.445595:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827287.445598:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517712000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827287.445631:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517712128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827287.445675:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827287.445688:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827287/real 1787827287] req@ffff8b2e88038f00 x1874672517712128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827287.445699:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517712128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827287.445707:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827287.445709:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827287.445712:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517712128:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827288.276858:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827288.468780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827288.468780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827289.300924:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827289.492769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827289.492787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827290.324925:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827290.516765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827290.516855:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827291.348814:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827291.540715:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827291.540759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827292.372828:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827292.564771:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827292.564781:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827292.564786:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827292.564793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787827292.564797:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 298 00000100:00080000:2.0:1787827292.564809:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827292.564816:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827292.564846:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517712256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827292.564867:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827292.564869:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827292.564871:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827292.564874:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 298 00000100:00080000:2.0:1787827292.564877:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827292.564880:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827292.564889:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827292.564893:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517712256:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827292.564906:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517712384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827292.564919:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827292.564922:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827292.564924:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827292.564928:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 298 00000100:00080000:2.0:1787827292.564944:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827292.564949:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827292.564959:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827292.564973:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517712512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827292.564992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827292.564996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827292.564998:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827292.565002:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 298 00000100:00080000:2.0:1787827292.565006:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827292.565010:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827292.565029:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827292.565038:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517712384:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827292.565046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517712640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827292.565059:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827292.565063:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827292.565065:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827292.565070:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 298 00000100:00080000:2.0:1787827292.565074:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827292.565079:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827292.565082:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517712512:192.168.203.101@tcp:8:kworker.0 00000008:00100000:2.0:1787827292.565086:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827292.565104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517712768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827292.565115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827292.565118:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827292.565121:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827292.565125:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517712640:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787827292.565130:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 298 00000100:00080000:2.0:1787827292.565136:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827292.565141:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827292.565146:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827292.565159:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517712768:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827292.565163:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517712896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827292.565177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827292.565181:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827292.565183:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827292.565187:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 298 00000100:00080000:2.0:1787827292.565192:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827292.565193:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517712896:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787827292.565200:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827292.565207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827292.565221:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517713024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827292.565271:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517713024:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827293.396798:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827293.588740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827293.589078:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827292/real 1787827293] req@ffff8b2e8801d680 x1874672517712640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.589116:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517712640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.589130:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.589135:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827293.589140:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517712640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827293.589147:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827292/real 1787827293] req@ffff8b2e8801cb40 x1874672517712768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.589160:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517712768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.589167:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.589172:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827293.589175:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517712768:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827293.589182:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827292/real 1787827293] req@ffff8b2e8801c3c0 x1874672517712384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.589193:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517712384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.589201:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.589205:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827293.589208:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517712384:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827293.589215:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827292/real 1787827293] req@ffff8b2e8801de00 x1874672517712512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.589227:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517712512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.589235:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.589238:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827293.589242:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517712512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827293.589249:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827292/real 1787827293] req@ffff8b2e8801f0c0 x1874672517712256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.589263:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517712256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.589271:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.589275:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827293.589278:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517712256:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827293.589307:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827292/real 1787827293] req@ffff8b2e8801ed00 x1874672517713024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.589320:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517713024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.589328:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.589332:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827293.589335:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517713024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827293.589359:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827292/real 1787827293] req@ffff8b2e8801c780 x1874672517712896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827293.589374:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517712896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827293.589382:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827293.589386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827293.589389:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517712896:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827294.420794:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827294.612749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827295.444807:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827295.636707:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827296.468809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827296.660735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827297.492971:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827297.684772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827297.684773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787827297.684784:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827297.684788:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827297.684799:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 304 00000100:00080000:2.0:1787827297.684811:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827297.684837:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827297.684872:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517713152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827297.684912:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827297.684916:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827297.684917:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827297.684921:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 304 00000100:00080000:2.0:1787827297.684925:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827297.684929:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827297.684938:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827297.684952:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517713280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827297.684962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827297.684968:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787827297.684970:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517713152:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787827297.684980:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827297.684986:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 304 00000100:00080000:2.0:1787827297.684989:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827297.684991:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827297.684995:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827297.685004:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d2c0 x1874672517713408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827297.685011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827297.685013:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827297.685015:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827297.685017:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 304 00000100:00080000:2.0:1787827297.685020:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827297.685023:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827297.685026:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827297.685033:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517713536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827297.685040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827297.685042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827297.685043:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827297.685046:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 304 00000100:00080000:2.0:1787827297.685048:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827297.685051:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827297.685054:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827297.685064:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517713664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827297.685070:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827297.685079:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827297.685081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827297.685087:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827297.685095:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 304 00000100:00080000:2.0:1787827297.685100:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827297.685105:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827297.685122:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827297.685141:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517713792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827297.685159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827297.685165:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827297/real 1787827297] req@ffff8b2e8801f840 x1874672517713152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827297.685165:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827297.685171:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827297.685192:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 304 00000100:00080000:2.0:1787827297.685201:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827297.685206:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827297.685213:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f840 x1874672517713152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827297.685216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827297.685242:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787827297.685244:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517713920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827297.685257:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827297.685270:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517713152:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827297.685316:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517713280:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827297.685349:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827297.685376:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827297/real 1787827297] req@ffff8b2e8801e1c0 x1874672517713280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827297.685393:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e1c0 x1874672517713280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827297.685404:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827297.685409:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827297.685414:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517713280:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827297.685423:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517713408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827297.685450:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827297.685465:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827297/real 1787827297] req@ffff8b2e8801d2c0 x1874672517713408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827297.685481:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d2c0 x1874672517713408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827297.685492:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827297.685496:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827297.685500:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517713408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827297.685508:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517713536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827297.685531:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827297.685544:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827297/real 1787827297] req@ffff8b2e8801e580 x1874672517713536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827297.685559:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517713536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827297.685569:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827297.685574:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827297.685578:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517713536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827297.685586:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517713664:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827297.685607:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827297.685620:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827297/real 1787827297] req@ffff8b2e8801e940 x1874672517713664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827297.685635:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517713664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827297.685673:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827297.685679:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827297.685683:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517713664:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827297.685691:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517713792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827297.685716:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827297.685730:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827297/real 1787827297] req@ffff8b2e8801f480 x1874672517713792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827297.685754:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517713792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827297.685763:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827297.685767:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827297.685771:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517713792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827297.685779:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517713920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827297.685801:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827297.685815:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827297/real 1787827297] req@ffff8b2e8801fc00 x1874672517713920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827297.685830:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517713920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827322 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827297.685840:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827297.685844:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827297.685849:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517713920:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827298.516977:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827298.708733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827299.540813:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827299.732709:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827300.564868:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827300.756790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827300.756813:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827301.589047:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827301.780727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827301.780727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827302.612829:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827302.804778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827302.804794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827302.804963:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517714048:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787827302.805060:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827302.805120:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827302/real 1787827302] req@ffff8b2e88039a40 x1874672517714048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827302.805169:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517714048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827302.805208:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827302.805218:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827302.805229:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517714048:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827302.805278:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517714176:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827302.805332:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827302.805357:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827302/real 1787827302] req@ffff8b2e88039680 x1874672517714176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827302.805407:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517714176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827302.805429:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827302.805439:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827302.805443:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517714176:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827302.805452:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517714304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827302.805497:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827302.805518:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827302/real 1787827302] req@ffff8b2e88039e00 x1874672517714304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827302.805533:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517714304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827302.805543:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827302.805548:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827302.805551:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517714304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827302.805559:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517714432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827302.805582:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827302.805595:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827302/real 1787827302] req@ffff8b2e880383c0 x1874672517714432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827302.805608:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517714432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827302.805617:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827302.805621:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827302.805625:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517714432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827302.805632:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517714560:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827302.805693:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827302.805714:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827302/real 1787827302] req@ffff8b2e8803b0c0 x1874672517714560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827302.805734:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517714560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827302.805752:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827302.805756:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827302.805759:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517714560:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827302.805781:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517714688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827302.805802:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827302.805813:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827302/real 1787827302] req@ffff8b2e88038780 x1874672517714688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827302.805824:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038780 x1874672517714688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827302.805833:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827302.805837:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827302.805841:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517714688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827302.805848:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517714816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827302.805884:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827302.805897:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827302/real 1787827302] req@ffff8b2e88038b40 x1874672517714816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827302.805910:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038b40 x1874672517714816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827327 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827302.805918:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827302.805922:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827302.805926:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517714816:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827303.636825:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827303.828746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827303.828746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827304.660870:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827304.852741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827304.852741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827305.684802:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827305.876729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827305.876818:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827306.708851:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827306.900742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827306.900749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827307.732858:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827307.924744:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827307.924750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787827307.924754:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827307.924759:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827307.924768:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 314 00000100:00080000:1.0:1787827307.924777:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827307.924783:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827307.924795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827307.924814:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517714944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827307.924854:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827307.924857:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827307.924859:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827307.924863:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 314 00000100:00080000:1.0:1787827307.924867:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827307.924870:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827307.924880:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827307.924894:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517714944:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827307.924897:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517715072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827307.924913:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827307.924916:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827307.924918:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827307.924922:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 314 00000100:00080000:1.0:1787827307.924926:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827307.924931:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827307.924937:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827307.924953:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517715200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827307.924965:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827307.924966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827307.924972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827307.924974:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827307.924980:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 314 00000100:00080000:1.0:1787827307.924987:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827307.924992:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827307.924998:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827307.925012:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827307/real 1787827307] req@ffff8b2e880752c0 x1874672517714944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827307.925015:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517715328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827307.925041:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827307.925044:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827307.925048:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827307.925048:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517714944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827307.925059:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 314 00000100:00080000:1.0:1787827307.925068:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827307.925078:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787827307.925082:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787827307.925086:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827307.925090:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827307.925097:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517714944:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827307.925106:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517715456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827307.925118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827307.925122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827307.925124:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827307.925129:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517715072:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827307.925132:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 314 00000100:00080000:1.0:1787827307.925140:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827307.925145:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827307.925152:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827307.925164:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827307.925171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517715584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827307.925183:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827307/real 1787827307] req@ffff8b2e88075a40 x1874672517715072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827307.925184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827307.925193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827307.925197:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827307.925208:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 314 00000100:00100000:2.0:1787827307.925213:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517715072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827307.925214:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827307.925225:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787827307.925230:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787827307.925233:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827307.925239:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827307.925244:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517715072:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827307.925251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517715712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827307.925260:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517715200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827307.925296:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827307.925310:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827307/real 1787827307] req@ffff8b2e88077c00 x1874672517715200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827307.925325:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517715200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827307.925335:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827307.925338:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827307.925342:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517715200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827307.925350:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517715328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827307.925370:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827307.925381:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827307/real 1787827307] req@ffff8b2e88077480 x1874672517715328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827307.925394:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517715328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827307.925402:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827307.925407:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827307.925411:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517715328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827307.925417:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517715456:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827307.925436:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827307.925448:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827307/real 1787827307] req@ffff8b2e88076940 x1874672517715456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827307.925461:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517715456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827307.925471:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827307.925474:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827307.925478:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517715456:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827307.925495:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517715584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827307.925514:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827307.925526:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827307/real 1787827307] req@ffff8b2e880752c0 x1874672517715584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827307.925540:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517715584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827307.925549:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827307.925552:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827307.925555:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517715584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827307.925562:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517715712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827307.925588:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827307.925599:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827307/real 1787827307] req@ffff8b2e880743c0 x1874672517715712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827307.925613:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517715712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827332 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827307.925621:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827307.925625:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827307.925628:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517715712:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827308.756867:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827308.948772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827309.780892:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827309.972805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827310.804829:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827310.996743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827311.828822:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827312.020766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827312.852867:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827313.044738:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827313.044745:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827313.044749:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827313.044761:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 319 00000100:00100000:2.0:1787827313.044769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827313.044771:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827313.044785:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827313.044837:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517715840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827313.044861:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827313.044864:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827313.044866:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827313.044870:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 319 00000100:00080000:1.0:1787827313.044878:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827313.044885:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827313.044898:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827313.044902:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517715840:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827313.044929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517715968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827313.044944:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827313.044949:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827313.044952:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827313.044957:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 319 00000100:00080000:1.0:1787827313.044961:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827313.044965:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827313.044971:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827313.044984:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517716096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827313.044995:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827313.044998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827313.045000:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827313.045005:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 319 00000100:00080000:1.0:1787827313.045010:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787827313.045014:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827313.045020:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827313.045028:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827313.045044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517716224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827313.045056:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827313.045060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827313.045062:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827313.045068:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 319 00000100:00080000:1.0:1787827313.045075:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827313.045081:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827313.045087:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827313/real 1787827313] req@ffff8b2e88074780 x1874672517715840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827313.045092:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827313.045122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517716352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827313.045130:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517715840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827313.045178:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827313.045179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827313.045193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827313.045195:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787827313.045199:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827313.045205:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517715840:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827313.045214:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 319 00000100:00080000:1.0:1787827313.045221:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827313.045227:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827313.045234:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827313.045253:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517715968:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827313.045254:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517716480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827313.045283:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827313.045286:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827313.045288:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827313.045294:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 319 00000100:00080000:1.0:1787827313.045299:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827313.045304:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827313.045311:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827313.045325:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827313.045334:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517716608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827313.045356:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827313/real 1787827313] req@ffff8b2e88074f00 x1874672517715968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827313.045376:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517715968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827313.045387:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827313.045391:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827313.045394:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517715968:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827313.045402:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517716096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827313.045430:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827313.045443:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827313/real 1787827313] req@ffff8b2e88075680 x1874672517716096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827313.045456:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517716096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827313.045470:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827313.045473:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827313.045477:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517716096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827313.045484:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517716224:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827313.045506:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827313.045518:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827313/real 1787827313] req@ffff8b2e88077840 x1874672517716224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827313.045529:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517716224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827313.045537:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827313.045540:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827313.045544:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517716224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827313.045553:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517716352:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827313.045573:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827313.045583:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827313/real 1787827313] req@ffff8b2e88074000 x1874672517716352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827313.045603:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517716352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827313.045612:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827313.045616:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827313.045621:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517716352:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827313.045687:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517716480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827313.045710:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827313.045724:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827313/real 1787827313] req@ffff8b2e88075e00 x1874672517716480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827313.045735:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517716480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827313.045744:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827313.045747:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827313.045751:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517716480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827313.045758:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517716608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827313.045780:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827313.045810:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827313/real 1787827313] req@ffff8b2e88076d00 x1874672517716608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827313.045824:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517716608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827313.045831:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827313.045835:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827313.045838:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517716608:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827313.876799:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827314.068738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827314.900917:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827315.092758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827315.924884:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827316.116811:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827316.116856:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827316.948834:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827317.140716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827317.140786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827317.972894:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827318.164740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827318.164951:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827318.164959:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827318.164965:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827318.164978:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 324 00000100:00080000:2.0:1787827318.164991:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827318.165000:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827318.165047:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517716736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827318.165074:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827318.165077:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827318.165079:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827318.165083:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 324 00000100:00080000:2.0:1787827318.165085:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827318.165088:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827318.165101:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827318.165110:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517716864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827318.165120:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827318.165127:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787827318.165130:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517716736:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787827318.165130:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827318.165139:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 324 00000100:00080000:2.0:1787827318.165145:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827318.165149:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827318.165154:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827318.165162:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517716992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827318.165169:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827318.165171:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827318.165173:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827318.165176:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 324 00000100:00080000:2.0:1787827318.165179:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827318.165181:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827318.165184:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827318.165193:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517717120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827318.165205:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827318.165208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827318.165210:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827318.165213:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 324 00000100:00080000:2.0:1787827318.165216:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827318.165225:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:1.0:1787827318.165230:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787827318.165233:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827318.165243:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517717248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827318.165252:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827318.165255:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827318.165257:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827318.165260:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 324 00000100:00080000:2.0:1787827318.165263:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827318.165267:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827318.165271:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827318.165284:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517717376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827318.165289:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827318/real 1787827318] req@ffff8b2e8801cf00 x1874672517716736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827318.165300:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827318.165309:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827318.165312:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827318.165328:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 324 00000100:00080000:2.0:1787827318.165333:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827318.165336:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827318.165339:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827318.165352:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517717504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827318.165368:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672517716736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827318.165383:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827318.165389:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827318.165395:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517716736:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827318.165420:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517716864:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827318.165448:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827318.165463:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827318/real 1787827318] req@ffff8b2e8801c000 x1874672517716864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827318.165477:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672517716864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827318.165486:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827318.165489:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827318.165493:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517716864:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827318.165500:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517716992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827318.165522:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827318.165535:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827318/real 1787827318] req@ffff8b2e8801da40 x1874672517716992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827318.165548:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672517716992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827318.165556:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827318.165560:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827318.165563:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517716992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827318.165570:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517717120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827318.165591:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827318.165602:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827318/real 1787827318] req@ffff8b2e8801fc00 x1874672517717120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827318.165614:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517717120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827318.165623:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827318.165626:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827318.165630:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517717120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827318.165636:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517717248:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827318.165683:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827318.165701:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827318/real 1787827318] req@ffff8b2e8801f480 x1874672517717248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827318.165722:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517717248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827318.165730:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827318.165733:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827318.165737:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517717248:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827318.165744:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517717376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827318.165782:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827318.165795:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827318/real 1787827318] req@ffff8b2e8801e940 x1874672517717376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827318.165807:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517717376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827318.165815:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827318.165818:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827318.165822:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517717376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827318.165829:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517717504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827318.165848:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827318.165858:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827318/real 1787827318] req@ffff8b2e8801e580 x1874672517717504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827318.165869:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517717504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827318.165882:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827318.165885:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827318.165888:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517717504:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827318.996863:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827319.188722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827320.020916:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827320.212832:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827321.044876:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827322.068852:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827323.092804:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827323.284795:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827323.284803:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827323.284808:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827323.284819:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 329 00000100:00080000:1.0:1787827323.284831:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827323.284837:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827323.284870:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517717632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827323.284896:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827323.284898:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827323.284900:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827323.284903:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 329 00000100:00080000:1.0:1787827323.284905:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827323.284908:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827323.284917:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827323.284926:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517717760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827323.284936:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827323.284942:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827323.284945:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827323.284948:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 329 00000100:00100000:2.0:1787827323.284951:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517717632:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827323.284952:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827323.284957:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827323.284969:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827323.284978:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517717888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827323.284984:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827323.284987:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827323.284988:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827323.284992:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 329 00000100:00080000:1.0:1787827323.284995:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827323.284998:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827323.285001:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827323.285008:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517718016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827323.285013:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827323.285015:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827323.285017:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827323.285020:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 329 00000100:00080000:1.0:1787827323.285022:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827323.285025:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827323.285030:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827323.285040:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517718144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827323.285051:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827323.285061:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827323.285063:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827323.285066:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827323.285073:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 329 00000100:00080000:1.0:1787827323.285077:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827323.285080:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827323.285084:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827323.285093:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517718272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827323.285100:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827323.285102:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827323.285104:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827323.285110:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 329 00000100:00080000:1.0:1787827323.285112:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827323.285114:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827323/real 1787827323] req@ffff8b2e880761c0 x1874672517717632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827323.285118:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827323.285130:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827323.285143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517718400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827323.285144:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517717632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827323.285183:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827323.285193:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827323.285199:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517717632:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827323.285226:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517717760:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827323.285244:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827323.285253:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827323/real 1787827323] req@ffff8b2e88076580 x1874672517717760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827323.285262:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517717760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827323.285268:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827323.285271:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827323.285273:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517717760:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827323.285277:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517717888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827323.285288:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827323.285294:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827323/real 1787827323] req@ffff8b2e880770c0 x1874672517717888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827323.285301:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517717888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827323.285307:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827323.285309:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827323.285311:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517717888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827323.285315:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517718016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827323.285325:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827323.285331:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827323/real 1787827323] req@ffff8b2e88074b40 x1874672517718016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827323.285339:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517718016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827323.285344:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827323.285345:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827323.285347:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517718016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827323.285351:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517718144:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827323.285361:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827323.285366:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827323/real 1787827323] req@ffff8b2e88076d00 x1874672517718144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827323.285373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517718144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827323.285379:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827323.285381:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827323.285382:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517718144:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827323.285386:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517718272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827323.285395:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827323.285401:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827323/real 1787827323] req@ffff8b2e88075e00 x1874672517718272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827323.285413:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517718272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827323.285418:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827323.285420:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827323.285422:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517718272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827323.285425:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517718400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827323.285435:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827323.285440:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827323/real 1787827323] req@ffff8b2e88074000 x1874672517718400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827323.285446:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517718400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827323.285451:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827323.285454:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827323.285456:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517718400:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827324.116822:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827325.140808:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827326.164818:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827327.188842:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827328.212861:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827328.404796:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827328.404809:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827328.404816:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827328.404830:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 334 00000100:00080000:1.0:1787827328.404845:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827328.404853:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827328.404901:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517718528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827328.404934:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827328.404937:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827328.404938:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827328.404942:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 334 00000100:00080000:1.0:1787827328.404946:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827328.404949:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827328.404959:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827328.404972:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517718656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827328.404984:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517718528:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827328.404995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827328.405000:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827328.405001:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827328.405004:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 334 00000100:00080000:1.0:1787827328.405006:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827328.405009:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827328.405012:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827328.405019:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517718784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827328.405026:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827328.405029:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827328.405030:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827328.405033:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 334 00000100:00080000:1.0:1787827328.405035:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827328.405038:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827328.405041:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827328.405058:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517718912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827328.405064:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827328.405067:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827328.405068:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827328.405072:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 334 00000100:00080000:1.0:1787827328.405074:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827328.405078:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827328.405084:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827328.405096:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517719040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827328.405108:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827328.405113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827328.405116:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827328.405122:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 334 00000100:00080000:1.0:1787827328.405127:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827328.405133:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517718656:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827328.405134:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827328.405143:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827328.405159:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517719168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827328.405178:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827328.405181:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827328.405183:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827328.405188:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 334 00000100:00100000:2.0:1787827328.405189:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517718784:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827328.405196:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827328.405201:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827328.405206:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827328.405217:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517718912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827328.405218:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517719296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827328.405240:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517719040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827328.405259:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517719168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827328.405274:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517719296:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827329.236863:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827330.260864:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827331.284857:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827331.477246:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787827331.477302:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827331.477326:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827331.477378:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827331.477396:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827331.477409:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827331.477418:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827331.477429:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827331.477441:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:1.0:1787827331.477443:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827328/real 1787827331] req@ffff8b2e88075680 x1874672517718656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827331.477477:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517718656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827331.477492:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827331.477498:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827331.477503:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517718656:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827331.477512:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827328/real 1787827331] req@ffff8b2e88074f00 x1874672517718784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827331.477527:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517718784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827331.477535:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827331.477540:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827331.477543:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517718784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827331.477550:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827328/real 1787827331] req@ffff8b2e88074780 x1874672517718912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827331.477561:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517718912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827331.477568:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827331.477572:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827331.477575:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517718912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827331.477582:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827328/real 1787827331] req@ffff8b2e880743c0 x1874672517719040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827331.477596:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517719040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827331.477604:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827331.477608:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827331.477611:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517719040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827331.477619:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827328/real 1787827331] req@ffff8b2e88077840 x1874672517718528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827331.477632:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517718528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827331.477640:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827331.477644:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827331.477687:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517718528:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827331.477726:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827328/real 1787827331] req@ffff8b2e880752c0 x1874672517719168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827331.477742:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517719168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827331.477759:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827331.477768:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827331.477772:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517719168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827331.477779:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827328/real 1787827331] req@ffff8b2e88076940 x1874672517719296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827331.477792:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517719296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827331.477800:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827331.477804:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827331.477807:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517719296:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827332.308857:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827332.500773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827333.332854:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827333.524770:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827333.524771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827333.524782:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827333.524787:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827333.524798:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 339 00000100:00080000:2.0:1787827333.524809:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827333.524817:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827333.524860:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d2c0 x1874672517719424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827333.524890:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827333.524895:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827333.524897:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827333.524907:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 339 00000100:00080000:2.0:1787827333.524911:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827333.524916:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827333.524928:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517719424:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787827333.524928:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827333.524948:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517719552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827333.524961:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827333.524965:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827333.524967:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827333.524972:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 339 00000100:00080000:2.0:1787827333.524977:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827333.524982:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827333.524987:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827333.525000:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517719680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827333.525011:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827333.525016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827333.525019:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:1.0:1787827333.525021:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827333.525026:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 339 00000100:00080000:2.0:1787827333.525030:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827333.525042:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827333.525049:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827333.525064:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517719808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827333.525076:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827333.525082:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff8b2e8801d2c0 x1874672517719424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827333.525083:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827333.525088:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827333.525097:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 339 00000100:00080000:2.0:1787827333.525104:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827333.525109:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827333.525115:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827333.525132:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517719936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827333.525133:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d2c0 x1874672517719424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827333.525161:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827333.525165:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827333.525166:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827333.525169:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827333.525174:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 339 00000100:00080000:2.0:1787827333.525182:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827333.525186:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787827333.525190:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827333.525196:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517719424:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787827333.525200:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827333.525216:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517720064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827333.525226:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827333.525230:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827333.525233:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827333.525239:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517719552:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827333.525239:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 339 00000100:00080000:2.0:1787827333.525246:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827333.525252:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827333.525259:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827333.525273:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517720192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827333.525282:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827333.525317:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff8b2e8801e1c0 x1874672517719552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827333.525333:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e1c0 x1874672517719552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827333.525343:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827333.525347:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827333.525351:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517719552:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827333.525358:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517719680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827333.525398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827333.525411:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff8b2e8801f840 x1874672517719680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827333.525425:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f840 x1874672517719680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827333.525434:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827333.525438:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827333.525442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517719680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827333.525449:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517719808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827333.525470:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827333.525486:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff8b2e8801c780 x1874672517719808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827333.525505:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517719808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827333.525514:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827333.525518:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827333.525521:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517719808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827333.525527:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517719936:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827333.525548:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827333.525562:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff8b2e8801ed00 x1874672517719936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827333.525573:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517719936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827333.525589:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827333.525592:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827333.525595:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517719936:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827333.525602:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517720064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827333.525623:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827333.525635:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff8b2e8801f0c0 x1874672517720064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827333.525687:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517720064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827333.525696:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827333.525699:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827333.525703:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517720064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827333.525725:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517720192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827333.525748:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827333.525759:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827333/real 1787827333] req@ffff8b2e8801de00 x1874672517720192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827333.525772:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517720192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827333.525781:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827333.525785:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827333.525788:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517720192:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827334.356877:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827334.548767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827335.572757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827336.404832:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827336.596778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827337.428865:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827337.620800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827338.452859:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827338.644790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787827338.644814:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827338.644824:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827338.644829:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827338.644840:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 345 00000100:00080000:1.0:1787827338.644851:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827338.644859:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827338.644899:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517720320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827338.644928:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827338.644933:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827338.644935:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827338.644941:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 345 00000100:00080000:1.0:1787827338.644946:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827338.644951:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827338.644962:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827338.644963:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517720320:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827338.644988:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517720448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827338.645003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827338.645007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827338.645010:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827338.645014:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 345 00000100:00080000:1.0:1787827338.645019:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827338.645025:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827338.645032:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827338.645046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517720576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827338.645058:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827338.645070:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827338.645075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827338.645077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827338.645083:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 345 00000100:00080000:1.0:1787827338.645088:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827338.645095:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827338.645101:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827338.645111:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff8b2e88076940 x1874672517720320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827338.645121:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517720704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827338.645142:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827338.645146:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787827338.645150:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517720320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827338.645151:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827338.645171:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 345 00000100:00080000:2.0:1787827338.645175:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827338.645178:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827338.645182:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787827338.645191:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827338.645194:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517720320:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787827338.645201:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827338.645220:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517720832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827338.645223:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517720448:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827338.645250:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827338.645256:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:2.0:1787827338.645261:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827338.645263:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827338.645271:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 345 00000100:00080000:1.0:1787827338.645277:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827338.645282:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827338.645287:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff8b2e880752c0 x1874672517720448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827338.645291:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827338.645316:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517720960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827338.645319:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517720448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827338.645371:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827338.645385:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827338.645399:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517720448:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827338.645405:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827338.645411:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517720576:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827338.645412:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827338.645419:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827338.645429:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 345 00000100:00080000:1.0:1787827338.645436:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827338.645442:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:2.0:1787827338.645445:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787827338.645455:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827338.645460:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff8b2e88077840 x1874672517720576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827338.645477:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517720576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827338.645482:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517721088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827338.645509:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827338.645514:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827338.645518:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517720576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827338.645524:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517720704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827338.645539:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827338.645547:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff8b2e880743c0 x1874672517720704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827338.645555:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517720704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827338.645560:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827338.645563:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827338.645565:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517720704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827338.645578:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517720832:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827338.645588:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827338.645594:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff8b2e88074780 x1874672517720832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827338.645602:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517720832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827338.645608:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827338.645610:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827338.645626:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517720832:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827338.645631:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517720960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827338.645666:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827338.645676:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff8b2e88074f00 x1874672517720960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827338.645684:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517720960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827338.645690:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827338.645692:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827338.645694:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517720960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827338.645698:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517721088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827338.645710:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827338.645716:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827338/real 1787827338] req@ffff8b2e88075680 x1874672517721088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827338.645724:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517721088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827338.645734:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827338.645736:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827338.645738:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517721088:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827339.476881:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827339.668825:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827340.692735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827341.524824:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827341.716770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827342.548890:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827342.740786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827343.572933:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827343.764775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827343.765096:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517721216:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827343.765189:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827343.765250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff8b2e880403c0 x1874672517721216/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827343.765301:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672517721216/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827343.765335:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827343.765348:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827343.765358:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517721216:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827343.765400:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517721344:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827343.765471:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827343.765504:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff8b2e88041e00 x1874672517721344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827343.765521:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517721344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827343.765532:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827343.765537:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827343.765541:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517721344:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827343.765566:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517721472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827343.765634:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827343.765675:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff8b2e88040b40 x1874672517721472/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827343.765687:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517721472/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827343.765695:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827343.765698:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827343.765700:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517721472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827343.765707:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517721600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827343.765733:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827343.765749:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff8b2e880421c0 x1874672517721600/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827343.765767:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517721600/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827343.765778:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827343.765783:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827343.765788:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517721600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827343.765795:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517721728:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827343.765816:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827343.765827:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff8b2e88042d00 x1874672517721728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827343.765837:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517721728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827343.765845:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827343.765848:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827343.765850:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517721728:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827343.765855:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517721856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827343.765867:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827343.765875:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff8b2e88043840 x1874672517721856/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827343.765884:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517721856/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827343.765894:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827343.765899:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827343.765904:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517721856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827343.765932:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517721984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827343.765961:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827343.765975:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827343/real 1787827343] req@ffff8b2e88040780 x1874672517721984/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827343.765992:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517721984/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827368 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827343.766003:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827343.766007:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827343.766012:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517721984:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827344.596854:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827344.788721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827345.812746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827346.644847:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827346.836811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827347.668883:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827347.860746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827348.692796:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827348.884793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787827348.884832:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827348.884857:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827348.884863:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827348.884876:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 355 00000100:00080000:1.0:1787827348.884891:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827348.884901:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827348.884959:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517722112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827348.884990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827348.884994:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827348.884995:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827348.884999:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 355 00000100:00080000:1.0:1787827348.885005:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827348.885011:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827348.885024:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827348.885026:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517722112:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827348.885054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517722240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827348.885067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827348.885070:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827348.885072:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827348.885075:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 355 00000100:00080000:1.0:1787827348.885080:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827348.885082:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827348.885087:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827348.885099:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517722368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827348.885108:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827348.885115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827348.885119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827348.885120:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827348.885124:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 355 00000100:00080000:1.0:1787827348.885129:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827348.885132:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827348.885135:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827348.885144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517722496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827348.885151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827348.885154:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827348.885156:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827348.885160:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 355 00000100:00080000:1.0:1787827348.885165:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827348.885165:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff8b2e88077480 x1874672517722112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827348.885172:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827348.885181:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827348.885192:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517722624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827348.885195:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517722112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827348.885214:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827348.885219:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827348.885220:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827348.885221:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827348.885230:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787827348.885231:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 355 00000100:00080000:1.0:1787827348.885237:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827348.885240:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517722112:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827348.885244:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827348.885251:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827348.885275:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517722752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827348.885292:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827348.885295:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517722240:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827348.885297:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827348.885302:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827348.885313:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 355 00000100:00080000:1.0:1787827348.885317:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827348.885321:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827348.885325:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827348.885336:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517722880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827348.885363:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827348.885382:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff8b2e88077c00 x1874672517722240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827348.885395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517722240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827348.885403:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827348.885407:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827348.885410:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517722240:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827348.885417:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517722368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827348.885451:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827348.885459:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff8b2e88075a40 x1874672517722368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827348.885467:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517722368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827348.885474:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827348.885476:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827348.885478:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517722368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827348.885482:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517722496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827348.885509:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827348.885518:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff8b2e88075680 x1874672517722496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827348.885528:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517722496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827348.885536:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827348.885539:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827348.885564:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517722496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827348.885570:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517722624:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827348.885619:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827348.885632:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff8b2e88074f00 x1874672517722624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827348.885644:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517722624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827348.885691:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827348.885694:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827348.885697:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517722624:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827348.885712:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517722752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827348.885728:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827348.885736:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff8b2e88074780 x1874672517722752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827348.885745:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517722752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827348.885751:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827348.885754:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827348.885756:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517722752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827348.885761:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517722880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827348.885772:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827348.885778:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827348/real 1787827348] req@ffff8b2e880743c0 x1874672517722880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827348.885786:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517722880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827373 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827348.885792:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827348.885793:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827348.885795:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517722880:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827349.716874:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827349.908785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827350.932772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827351.764878:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827351.956784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827352.788828:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827352.980800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827353.812863:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827354.004752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787827354.004771:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827354.004780:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827354.004785:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827354.004798:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 360 00000100:00080000:1.0:1787827354.004807:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827354.004814:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827354.004848:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517723008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827354.004873:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827354.004876:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827354.004888:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827354.004891:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 360 00000100:00080000:1.0:1787827354.004894:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827354.004896:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827354.004905:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827354.004916:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517723136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827354.004925:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827354.004929:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827354.004934:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827354.004934:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517723008:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827354.004952:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 360 00000100:00080000:1.0:1787827354.004960:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827354.004964:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827354.004969:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827354.004994:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517723264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827354.005001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827354.005003:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827354.005005:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827354.005008:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 360 00000100:00080000:1.0:1787827354.005010:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827354.005012:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827354.005016:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827354.005023:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517723392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827354.005030:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827354.005033:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827354.005035:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827354.005039:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 360 00000100:00080000:1.0:1787827354.005042:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827354.005044:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827354.005047:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827354.005052:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827354.005057:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517723520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827354.005065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827354.005067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827354.005069:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827354.005073:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 360 00000100:00080000:1.0:1787827354.005079:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827354.005082:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827354.005085:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827354.005090:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517723648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827354.005096:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827354.005098:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827354.005100:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827354.005104:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 360 00000100:00080000:1.0:1787827354.005108:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827354.005111:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827354.005117:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827354.005121:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827354/real 1787827354] req@ffff8b2e88077840 x1874672517723008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827354.005128:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517723776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827354.005173:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517723008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827354.005190:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827354.005196:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827354.005202:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517723008:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827354.005258:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517723136:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827354.005288:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827354.005303:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827354/real 1787827354] req@ffff8b2e880752c0 x1874672517723136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827354.005329:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517723136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827354.005336:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827354.005339:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827354.005341:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517723136:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827354.005346:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517723264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827354.005364:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827354.005372:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827354/real 1787827354] req@ffff8b2e88076940 x1874672517723264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827354.005381:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517723264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827354.005387:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827354.005389:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827354.005392:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517723264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827354.005396:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517723392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827354.005428:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827354.005437:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827354/real 1787827354] req@ffff8b2e88074000 x1874672517723392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827354.005446:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517723392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827354.005452:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827354.005455:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827354.005457:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517723392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827354.005462:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517723520:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827354.005490:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827354.005496:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827354/real 1787827354] req@ffff8b2e88075e00 x1874672517723520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827354.005504:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517723520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827354.005524:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827354.005527:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827354.005529:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517723520:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827354.005533:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517723648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827354.005567:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827354.005577:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827354/real 1787827354] req@ffff8b2e88076d00 x1874672517723648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827354.005586:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517723648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827354.005594:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827354.005596:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827354.005599:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517723648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827354.005604:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517723776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827354.005617:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827354.005624:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827354/real 1787827354] req@ffff8b2e88074b40 x1874672517723776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827354.005633:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517723776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827354.005640:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827354.005642:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827354.005644:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517723776:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827354.836862:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827355.028815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827356.052760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827356.884807:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827357.076804:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827357.908845:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827358.100778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827358.932892:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827359.124783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827359.124816:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827359.124826:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827359.124832:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827359.124846:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 365 00000100:00080000:2.0:1787827359.124859:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827359.124866:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827359.124910:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517723904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827359.124940:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827359.124944:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827359.124947:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827359.124951:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 365 00000100:00080000:2.0:1787827359.124955:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827359.124960:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827359.124972:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827359.124987:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517724032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827359.124996:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827359.124999:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517723904:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787827359.125000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827359.125003:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827359.125012:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 365 00000100:00080000:2.0:1787827359.125014:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827359.125017:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827359.125022:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827359.125032:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517724160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827359.125039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827359.125042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827359.125043:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827359.125045:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 365 00000100:00080000:2.0:1787827359.125048:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827359.125051:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827359.125054:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827359.125062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517724288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827359.125067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827359.125069:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827359.125070:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827359.125073:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 365 00000100:00080000:2.0:1787827359.125076:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827359.125082:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827359.125082:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827359.125088:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827359.125096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517724416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827359.125106:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827359.125110:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827359.125112:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827359.125116:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 365 00000100:00080000:2.0:1787827359.125120:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827359.125125:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827359.125131:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827359.125140:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827359/real 1787827359] req@ffff8b2e8801c3c0 x1874672517723904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827359.125146:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517724544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827359.125164:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517723904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827359.125165:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827359.125172:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827359.125175:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827359.125181:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 365 00000100:00080000:1.0:1787827359.125185:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827359.125186:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827359.125192:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:1.0:1787827359.125194:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:2.0:1787827359.125198:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827359.125200:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517723904:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827359.125211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517724672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827359.125232:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517724032:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827359.125260:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827359.125272:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827359/real 1787827359] req@ffff8b2e8801cb40 x1874672517724032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827359.125282:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517724032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827359.125289:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827359.125291:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827359.125293:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517724032:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827359.125298:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517724160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827359.125312:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827359.125318:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827359/real 1787827359] req@ffff8b2e8801d680 x1874672517724160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827359.125326:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517724160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827359.125332:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827359.125333:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827359.125335:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517724160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827359.125339:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517724288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827359.125348:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827359.125354:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827359/real 1787827359] req@ffff8b2e8801de00 x1874672517724288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827359.125361:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517724288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827359.125371:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827359.125373:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827359.125376:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517724288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827359.125379:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517724416:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827359.125388:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827359.125394:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827359/real 1787827359] req@ffff8b2e8801f0c0 x1874672517724416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827359.125401:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517724416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827359.125406:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827359.125408:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827359.125410:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517724416:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827359.125413:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517724544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827359.125423:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827359.125429:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827359/real 1787827359] req@ffff8b2e8801ed00 x1874672517724544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827359.125436:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517724544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827359.125441:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827359.125442:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827359.125445:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517724544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827359.125448:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517724672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827359.125457:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827359.125462:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827359/real 1787827359] req@ffff8b2e8801c780 x1874672517724672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827359.125483:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517724672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827359.125489:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827359.125492:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827359.125494:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517724672:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827359.956795:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827360.148763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827361.172734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827362.004777:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827362.196751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827363.028833:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827363.220756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827364.052852:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827364.244741:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827364.244759:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827364.244766:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827364.244780:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 370 00000100:00100000:1.0:1787827364.244790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827364.244796:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827364.244812:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827364.244866:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517724800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827364.244899:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827364.244903:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827364.244905:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827364.244910:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 370 00000100:00080000:2.0:1787827364.244913:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827364.244917:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827364.244930:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827364.244943:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517724928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827364.244954:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517724800:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787827364.244973:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827364.244977:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827364.244979:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827364.244984:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 370 00000100:00080000:2.0:1787827364.244987:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827364.244990:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827364.244996:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827364.245006:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d2c0 x1874672517725056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827364.245014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827364.245017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827364.245019:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827364.245022:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 370 00000100:00080000:2.0:1787827364.245025:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827364.245028:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827364.245034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827364.245053:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827364.245055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517725184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827364.245066:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827364.245070:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827364.245073:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827364.245076:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 370 00000100:00080000:2.0:1787827364.245080:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827364.245084:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827364.245090:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827364.245105:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517725312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827364.245112:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff8b2e8801f840 x1874672517724800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827364.245141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827364.245155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787827364.245160:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f840 x1874672517724800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827364.245161:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827364.245173:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 370 00000100:00080000:2.0:1787827364.245202:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827364.245204:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827364.245212:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:1.0:1787827364.245213:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827364.245225:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517724800:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787827364.245228:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827364.245249:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517725440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827364.245256:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517724928:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827364.245287:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827364.245293:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827364.245296:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827364.245301:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 370 00000800:00000100:1.0:1787827364.245312:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827364.245313:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827364.245321:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827364.245328:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827364.245341:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff8b2e8801e1c0 x1874672517724928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827364.245344:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517725568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827364.245392:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e1c0 x1874672517724928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827364.245404:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827364.245409:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827364.245412:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517724928:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827364.245421:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517725056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827364.245458:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827364.245471:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff8b2e8801d2c0 x1874672517725056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827364.245484:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d2c0 x1874672517725056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827364.245504:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827364.245508:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827364.245512:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517725056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827364.245520:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517725184:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827364.245541:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827364.245554:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff8b2e8801e580 x1874672517725184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827364.245567:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517725184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827364.245576:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827364.245580:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827364.245583:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517725184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827364.245590:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517725312:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827364.245612:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827364.245624:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff8b2e8801e940 x1874672517725312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827364.245637:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517725312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827364.245672:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827364.245677:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827364.245681:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517725312:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827364.245701:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517725440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827364.245724:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827364.245737:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff8b2e8801f480 x1874672517725440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827364.245752:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517725440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827364.245760:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827364.245764:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827364.245767:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517725440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827364.245774:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517725568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827364.245796:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827364.245808:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827364/real 1787827364] req@ffff8b2e8801fc00 x1874672517725568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827364.245819:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517725568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827364.245828:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827364.245832:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827364.245835:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517725568:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827365.076820:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827365.268722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827366.292761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827367.124859:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827367.316740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827368.148794:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827368.340810:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827369.172872:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827369.364765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827369.364879:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827369.364892:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827369.364916:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827369.364934:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 375 00000100:00080000:2.0:1787827369.364956:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827369.364971:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827369.365069:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517725696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827369.365106:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827369.365110:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827369.365113:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827369.365119:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 375 00000100:00080000:2.0:1787827369.365125:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827369.365130:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827369.365153:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827369.365165:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517725696:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827369.365180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517725824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827369.365198:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827369.365203:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827369.365206:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827369.365211:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 375 00000100:00080000:2.0:1787827369.365215:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827369.365221:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827369.365229:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827369.365242:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517725952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827369.365251:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827369.365274:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827369.365280:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827369.365285:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827369.365294:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 375 00000100:00080000:2.0:1787827369.365303:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827369.365304:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827369] req@ffff8b2e8801da40 x1874672517725696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827369.365341:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827369.365351:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672517725696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827369.365358:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827369.365375:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827369.365383:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827369.365390:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517726080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827369.365397:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517725696:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787827369.365455:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827369.365456:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517725824:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827369.365473:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827369.365478:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827369.365503:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 375 00000800:00000100:1.0:1787827369.365514:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827369.365515:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827369.365532:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827369.365541:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827369.365544:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827369] req@ffff8b2e8801c000 x1874672517725824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827369.365568:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672517725824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827369.365572:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517726208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827369.365641:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827369.365652:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827369.365694:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827369.365701:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517725824:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827369.365711:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517725952:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787827369.365727:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827369.365748:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827369.365762:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 375 00000800:00000100:1.0:1787827369.365772:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827369.365781:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827369.365794:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827369.365800:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827369] req@ffff8b2e8801cf00 x1874672517725952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827369.365819:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827369.365872:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672517725952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827369.365875:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517726336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827369.365935:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827369.365942:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827369.365945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827369.365950:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827369.365960:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 375 00000100:00080000:2.0:1787827369.365986:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827369.365994:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787827369.366004:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827369.366014:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517725952:192.168.203.101@tcp:8:kworker.0 00000008:00100000:2.0:1787827369.366021:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827369.366050:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517726464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827369.366071:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517726080:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827369.366131:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827369.366157:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827369] req@ffff8b2e8801fc00 x1874672517726080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827369.366171:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517726080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827369.366198:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827369.366203:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827369.366207:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517726080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827369.366216:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517726208:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827369.366238:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827369.366250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827369] req@ffff8b2e8801f480 x1874672517726208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827369.366264:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517726208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827369.366283:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827369.366288:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827369.366291:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517726208:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827369.366299:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517726336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827369.366318:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827369.366329:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827369] req@ffff8b2e8801e940 x1874672517726336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827369.366358:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517726336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827369.366365:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827369.366368:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827369.366370:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517726336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827369.366384:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517726464:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827369.366420:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827369.366440:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827369/real 1787827369] req@ffff8b2e8801e580 x1874672517726464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827369.366450:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517726464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827369.366457:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827369.366459:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827369.366463:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517726464:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827370.196868:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827370.388740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827371.412786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827372.244934:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827372.436741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827373.268814:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827373.460763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827374.292831:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827374.484805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827374.484830:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827374.484837:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827374.484841:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827374.484855:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 380 00000100:00080000:2.0:1787827374.484867:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827374.484886:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827374.484924:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d2c0 x1874672517726592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827374.484960:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827374.484964:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827374.484966:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827374.484969:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 380 00000100:00080000:2.0:1787827374.484972:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827374.484975:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827374.484984:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827374.484995:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517726720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827374.485005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827374.485008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827374.485011:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827374.485015:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517726592:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787827374.485017:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 380 00000100:00080000:2.0:1787827374.485023:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827374.485030:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827374.485034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827374.485043:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517726848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827374.485051:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827374.485054:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827374.485055:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827374.485059:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 380 00000100:00080000:2.0:1787827374.485062:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827374.485065:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827374.485069:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827374.485078:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517726976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827374.485085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827374.485088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827374.485089:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827374.485094:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 380 00000100:00080000:2.0:1787827374.485098:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827374.485101:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827374.485105:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827374.485110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827374.485119:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517727104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827374.485127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827374.485130:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827374.485131:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827374.485133:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 380 00000100:00080000:2.0:1787827374.485136:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827374.485150:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827374.485153:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827374.485163:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517727232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827374.485174:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff8b2e8801d2c0 x1874672517726592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827374.485180:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827374.485186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827374.485189:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827374.485198:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 380 00000100:00080000:2.0:1787827374.485201:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827374.485203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827374.485207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827374.485238:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517727360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827374.485252:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d2c0 x1874672517726592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827374.485294:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827374.485303:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827374.485310:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517726592:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827374.485347:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517726720:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827374.485438:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827374.485472:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff8b2e8801e1c0 x1874672517726720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827374.485490:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e1c0 x1874672517726720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827374.485503:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827374.485508:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827374.485513:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517726720:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827374.485538:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517726848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827374.485567:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827374.485582:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff8b2e8801f840 x1874672517726848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827374.485598:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f840 x1874672517726848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827374.485610:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827374.485615:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827374.485619:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517726848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827374.485627:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517726976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827374.485653:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827374.485668:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff8b2e8801c780 x1874672517726976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827374.485715:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517726976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827374.485726:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827374.485730:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827374.485735:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517726976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827374.485744:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517727104:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827374.485782:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827374.485797:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff8b2e8801ed00 x1874672517727104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827374.485840:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517727104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827374.485852:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827374.485856:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827374.485860:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517727104:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827374.485903:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517727232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827374.485953:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827374.485969:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff8b2e8801f0c0 x1874672517727232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827374.485994:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517727232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827374.486005:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827374.486009:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827374.486015:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517727232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827374.486024:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517727360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827374.486053:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827374.486070:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827374/real 1787827374] req@ffff8b2e8801de00 x1874672517727360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827374.486086:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517727360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827374.486098:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827374.486103:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827374.486107:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517727360:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827375.316800:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827375.508723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827376.532780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827377.364820:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827377.556745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827378.388821:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827378.580745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827379.412844:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827379.604739:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827379.604744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827379.604747:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827379.604752:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827379.604762:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 386 00000100:00080000:2.0:1787827379.604772:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827379.604779:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827379.604818:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517727488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827379.604849:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827379.604854:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827379.604856:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827379.604860:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 386 00000100:00080000:2.0:1787827379.604864:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827379.604868:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827379.604878:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517727488:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787827379.604878:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827379.604895:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517727616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827379.604904:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827379.604907:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827379.604908:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827379.604911:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 386 00000100:00080000:2.0:1787827379.604913:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827379.604916:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827379.604919:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827379.604927:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517727744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827379.604934:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827379.604937:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827379.604938:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827379.604940:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 386 00000100:00080000:2.0:1787827379.604943:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787827379.604946:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827379.604950:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827379.604955:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827379.604963:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517727872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827379.604970:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827379.604972:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827379.604974:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827379.604978:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 386 00000100:00080000:2.0:1787827379.604981:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827379.604985:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827379.604989:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff8b2e8801d680 x1874672517727488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827379.604990:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827379.605004:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517728000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827379.605010:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517727488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827379.605024:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827379.605028:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827379.605029:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827379.605030:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827379.605036:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787827379.605038:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 386 00000100:00100000:1.0:1787827379.605042:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517727488:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787827379.605042:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827379.605047:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827379.605052:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827379.605060:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517728128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827379.605068:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827379.605069:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517727616:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827379.605072:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827379.605076:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827379.605079:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 386 00000100:00080000:2.0:1787827379.605081:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827379.605084:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827379.605088:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827379.605093:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827379.605101:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517728256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827379.605107:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff8b2e8801cb40 x1874672517727616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827379.605119:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517727616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827379.605126:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827379.605129:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827379.605132:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517727616:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827379.605137:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517727744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827379.605159:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827379.605165:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff8b2e8801c3c0 x1874672517727744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827379.605173:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517727744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827379.605178:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827379.605179:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827379.605181:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517727744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827379.605185:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517727872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827379.605194:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827379.605200:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff8b2e8801de00 x1874672517727872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827379.605207:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517727872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827379.605212:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827379.605214:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827379.605216:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517727872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827379.605226:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517728000:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827379.605235:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827379.605240:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff8b2e8801f0c0 x1874672517728000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827379.605247:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517728000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827379.605253:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827379.605254:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827379.605256:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517728000:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827379.605266:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517728128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827379.605276:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827379.605282:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff8b2e8801ed00 x1874672517728128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827379.605289:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517728128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827379.605298:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827379.605300:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827379.605302:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517728128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827379.605306:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517728256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827379.605314:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827379.605320:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827379/real 1787827379] req@ffff8b2e8801c780 x1874672517728256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827379.605339:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517728256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827379.605346:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827379.605348:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827379.605350:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517728256:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827380.436819:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827380.628733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827381.652719:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827382.484818:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827382.676750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827383.508814:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827383.700711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827384.532841:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827384.724764:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827384.724777:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827384.724783:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827384.724805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827384.724813:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 391 00000100:00080000:2.0:1787827384.724833:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827384.724847:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827384.724896:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517728384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827384.724924:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827384.724929:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827384.724931:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827384.724937:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 391 00000100:00080000:2.0:1787827384.724943:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827384.724949:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827384.724970:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517728384:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787827384.724974:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827384.725010:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517728512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827384.725025:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827384.725029:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827384.725032:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827384.725035:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 391 00000100:00080000:2.0:1787827384.725042:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827384.725049:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827384.725055:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827384.725076:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d2c0 x1874672517728640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827384.725084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827384.725087:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827384.725089:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827384.725093:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 391 00000100:00080000:2.0:1787827384.725096:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827384.725098:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827384.725102:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827384.725111:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827384.725114:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517728768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827384.725127:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827384.725130:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827384.725140:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827384.725145:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 391 00000100:00080000:2.0:1787827384.725149:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827384.725153:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827384.725159:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827384.725183:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517728896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827384.725189:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff8b2e8801f840 x1874672517728384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827384.725222:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827384.725244:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827384.725247:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827384.725258:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 391 00000100:00080000:2.0:1787827384.725264:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827384.725269:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f840 x1874672517728384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827384.725276:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827384.725307:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827384.725321:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517729024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827384.725371:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827384.725374:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827384.725377:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827384.725384:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787827384.725386:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827384.725392:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517728384:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787827384.725410:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 391 00000100:00080000:2.0:1787827384.725419:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827384.725421:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517728512:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827384.725438:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827384.725447:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827384.725460:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827384.725466:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517729152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827384.725481:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff8b2e8801e1c0 x1874672517728512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827384.725496:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e1c0 x1874672517728512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827384.725503:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827384.725507:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827384.725512:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517728512:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827384.725519:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517728640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827384.725536:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827384.725555:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff8b2e8801d2c0 x1874672517728640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827384.725565:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d2c0 x1874672517728640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827384.725595:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827384.725599:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827384.725603:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517728640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827384.725611:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517728768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827384.725632:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827384.725661:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff8b2e8801e580 x1874672517728768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827384.725673:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517728768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827384.725681:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827384.725684:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827384.725688:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517728768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827384.725755:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517728896:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827384.725791:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827384.725804:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff8b2e8801e940 x1874672517728896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827384.725815:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517728896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827384.725834:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827384.725837:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827384.725840:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517728896:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827384.725845:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517729024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827384.725861:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827384.725869:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff8b2e8801f480 x1874672517729024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827384.725878:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517729024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827384.725884:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827384.725886:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827384.725888:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517729024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827384.725905:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517729152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827384.725917:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827384.725925:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827384/real 1787827384] req@ffff8b2e8801fc00 x1874672517729152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827384.725934:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517729152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827409 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827384.725940:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827384.725942:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827384.725945:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517729152:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827385.556870:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827385.748803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827386.772801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827387.604856:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827387.796749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827388.628811:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827388.820744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827389.652847:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827389.844845:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827389.844873:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827389.844880:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827389.844893:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 396 00000100:00080000:1.0:1787827389.844908:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827389.844920:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827389.844980:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517729280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827389.845011:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827389.845016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827389.845018:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827389.845025:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 396 00000100:00080000:1.0:1787827389.845030:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827389.845035:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827389.845043:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517729280:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787827389.845048:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827389.845064:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517729408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827389.845078:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827389.845082:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827389.845085:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827389.845090:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 396 00000100:00080000:1.0:1787827389.845095:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827389.845100:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827389.845106:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827389.845120:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517729536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827389.845127:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827389.845137:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827389.845142:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827389.845145:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827389.845151:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 396 00000100:00080000:1.0:1787827389.845156:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827389.845161:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827389.845168:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827389.845194:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827389/real 1787827389] req@ffff8b2e880770c0 x1874672517729280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827389.845210:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517729664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827389.845225:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517729280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827389.845242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827389.845292:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827389.845299:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827389.845304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827389.845311:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 396 00000100:00080000:2.0:1787827389.845314:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827389.845323:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517729280:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827389.845324:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827389.845337:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827389.845346:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827389.845356:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517729408:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827389.845369:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517729792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827389.845387:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827389.845390:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827389.845397:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827389.845401:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827389.845404:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827389/real 1787827389] req@ffff8b2e88076580 x1874672517729408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827389.845412:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 396 00000100:00080000:1.0:1787827389.845425:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827389.845429:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517729408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827389.845438:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787827389.845443:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787827389.845450:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827389.845451:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827389.845458:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517729408:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827389.845467:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517729536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827389.845488:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827389.845491:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517729920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827389.845499:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827389/real 1787827389] req@ffff8b2e880761c0 x1874672517729536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827389.845531:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517729536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827389.845533:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827389.845548:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827389.845550:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827389.845555:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827389.845556:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827389.845562:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517729536:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827389.845571:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 396 00000100:00100000:2.0:1787827389.845572:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517729664:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827389.845585:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827389.845596:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:2.0:1787827389.845600:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787827389.845610:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827389.845614:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827389/real 1787827389] req@ffff8b2e88074b40 x1874672517729664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827389.845627:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517729664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827389.845635:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517730048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827389.845638:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827389.845683:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827389.845691:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517729664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827389.845722:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517729792:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827389.845751:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827389.845764:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827389/real 1787827389] req@ffff8b2e88076d00 x1874672517729792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827389.845774:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517729792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827389.845782:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827389.845785:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827389.845788:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517729792:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827389.845794:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517729920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827389.845823:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827389.845830:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827389/real 1787827389] req@ffff8b2e88075e00 x1874672517729920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827389.845839:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517729920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827389.845844:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827389.845845:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827389.845847:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517729920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827389.845851:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517730048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827389.845862:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827389.845868:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827389/real 1787827389] req@ffff8b2e88074000 x1874672517730048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827389.845875:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517730048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827414 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827389.845880:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827389.845882:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827389.845884:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517730048:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827390.676978:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827392.724832:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827392.916819:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827393.748855:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827393.940778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827394.772879:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827394.964791:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827394.964799:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827394.964805:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827394.964819:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 401 00000100:00100000:2.0:1787827394.964821:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827394.964835:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827394.964847:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827394.964904:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517730176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827394.964933:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827394.964937:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827394.964938:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827394.964943:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 401 00000100:00080000:1.0:1787827394.964947:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827394.964951:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827394.964963:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827394.964971:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517730176:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827394.964977:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517730304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827394.964988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827394.964991:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827394.964993:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827394.964996:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 401 00000100:00080000:1.0:1787827394.965000:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827394.965003:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827394.965008:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827394.965017:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517730432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827394.965027:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827394.965030:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827394.965031:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827394.965034:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 401 00000100:00080000:1.0:1787827394.965037:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827394.965040:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827394.965044:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827394.965051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517730560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827394.965059:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827394.965062:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827394.965064:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827394.965067:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 401 00000100:00080000:1.0:1787827394.965070:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827394.965073:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827394.965076:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827394.965085:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517730688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827394.965091:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827394.965095:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827394.965098:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827394.965103:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 401 00000100:00080000:1.0:1787827394.965107:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827394.965111:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827394.965115:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827394.965125:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517730304:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827394.965127:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517730816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827394.965142:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827394.965144:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827394.965146:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827394.965149:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 401 00000100:00080000:1.0:1787827394.965152:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827394.965155:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827394.965159:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827394.965171:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517730944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827394.965180:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517730432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827394.965206:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517730560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827394.965254:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517730688:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827394.965292:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517730816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827394.965318:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517730944:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827395.796905:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827395.988724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827397.012740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827397.844915:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827398.036807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:1.0:1787827398.037072:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787827398.037144:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827398.037175:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827398.037265:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827398.037285:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827398.037302:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827398.037322:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827398.037340:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:2.0:1787827398.037346:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827394/real 1787827398] req@ffff8b2e880752c0 x1874672517730304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787827398.037369:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:2.0:1787827398.037402:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517730304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827398.037426:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827398.037435:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827398.037442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517730304:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827398.037455:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827394/real 1787827398] req@ffff8b2e88077840 x1874672517730432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827398.037477:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517730432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827398.037490:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827398.037496:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827398.037501:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517730432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827398.037511:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827394/real 1787827398] req@ffff8b2e880743c0 x1874672517730560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827398.037529:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517730560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827398.037541:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827398.037546:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827398.037576:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517730560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827398.037587:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827394/real 1787827398] req@ffff8b2e88074780 x1874672517730688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827398.037608:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517730688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827398.037619:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827398.037625:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827398.037629:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517730688:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827398.037640:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827394/real 1787827398] req@ffff8b2e88076940 x1874672517730176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827398.037693:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517730176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827398.037707:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827398.037712:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827398.037718:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517730176:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827398.037761:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827394/real 1787827398] req@ffff8b2e88074f00 x1874672517730816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827398.037782:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517730816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827398.037795:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827398.037800:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827398.037805:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517730816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827398.037815:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827394/real 1787827398] req@ffff8b2e88075680 x1874672517730944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827398.037836:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517730944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827419 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827398.037850:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827398.037856:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827398.037861:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517730944:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827398.869001:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827399.060743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827399.892872:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827400.084806:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827400.084819:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827400.084834:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827400.084840:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827400.084868:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 406 00000100:00080000:1.0:1787827400.084887:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827400.084899:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827400.084992:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517731072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827400.085037:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827400.085044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827400.085048:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827400.085055:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 406 00000100:00080000:1.0:1787827400.085064:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827400.085084:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827400.085100:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827400.085113:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517731072:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827400.085144:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517731200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827400.085165:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827400.085172:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827400.085176:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827400.085205:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 406 00000100:00080000:1.0:1787827400.085215:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827400.085223:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:2.0:1787827400.085242:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787827400.085250:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827400.085277:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517731328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827400.085294:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827400.085299:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827400.085310:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827400.085311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827400/real 1787827400] req@ffff8b2e88075a40 x1874672517731072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827400.085326:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 406 00000100:00080000:1.0:1787827400.085338:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827400.085350:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827400.085360:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827400.085363:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517731072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827400.085385:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827400.085391:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517731456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827400.085399:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827400.085420:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517731072:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827400.085432:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827400.085440:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827400.085445:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827400.085454:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 406 00000100:00080000:1.0:1787827400.085461:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827400.085465:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517731200:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827400.085475:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827400.085488:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827400.085515:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517731584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827400.085526:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827400.085551:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827400.085556:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827400/real 1787827400] req@ffff8b2e88077c00 x1874672517731200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827400.085561:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827400.085569:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827400.085588:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 406 00000100:00080000:1.0:1787827400.085596:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827400.085598:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517731200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827400.085613:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787827400.085622:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787827400.085627:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827400.085635:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827400.085643:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517731200:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827400.085654:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517731712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827400.085685:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517731328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827400.085713:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827400.085728:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827400/real 1787827400] req@ffff8b2e88077480 x1874672517731328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827400.085762:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517731328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827400.085776:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827400.085781:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827400.085784:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827400.085791:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827400.085795:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827400.085807:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517731328:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827400.085815:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 406 00000100:00080000:1.0:1787827400.085826:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827400.085831:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517731456:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827400.085840:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827400.085852:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827400.085875:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827400.085881:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517731840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827400.085902:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827400/real 1787827400] req@ffff8b2e88075680 x1874672517731456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827400.085922:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517731456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827400.085933:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827400.085941:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827400.085945:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517731456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827400.085969:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517731584:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827400.085993:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827400.086007:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827400/real 1787827400] req@ffff8b2e88074f00 x1874672517731584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827400.086024:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517731584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827400.086034:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827400.086038:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827400.086042:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517731584:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827400.086049:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517731712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827400.086069:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827400.086082:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827400/real 1787827400] req@ffff8b2e88076940 x1874672517731712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827400.086102:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517731712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827400.086110:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827400.086114:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827400.086117:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517731712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827400.086124:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517731840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827400.086145:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827400.086158:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827400/real 1787827400] req@ffff8b2e88074780 x1874672517731840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827400.086171:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517731840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827400.086179:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827400.086182:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827400.086203:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517731840:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827400.916920:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827401.108724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827402.132736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827402.964819:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827403.156869:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827403.988899:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827404.180796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827405.012820:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827405.204765:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827405.204770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787827405.204784:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827405.204791:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827405.204806:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 411 00000100:00080000:2.0:1787827405.204819:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827405.204828:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827405.204876:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517731968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827405.204913:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827405.204917:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827405.204918:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827405.204922:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 411 00000100:00080000:2.0:1787827405.204926:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827405.204931:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827405.204943:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827405.204954:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517731968:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827405.204960:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517732096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827405.204971:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827405.204975:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827405.204977:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827405.204980:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 411 00000100:00080000:2.0:1787827405.204985:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827405.204988:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827405.204992:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827405.205002:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517732224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827405.205012:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827405.205016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827405.205019:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827405.205025:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 411 00000800:00000100:1.0:1787827405.205030:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827405.205039:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827405.205050:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827405.205057:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827405.205072:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517732352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827405.205076:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827405/real 1787827405] req@ffff8b2e8801cf00 x1874672517731968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827405.205120:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672517731968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827405.205125:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827405.205137:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827405.205138:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827405.205146:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827405.205148:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787827405.205171:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 411 00000100:00100000:1.0:1787827405.205175:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517731968:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787827405.205178:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827405.205186:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827405.205192:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827405.205205:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517732480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827405.205212:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517732096:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827405.205227:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827405.205231:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827405.205234:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827405.205240:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 411 00000100:00080000:2.0:1787827405.205245:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827405.205252:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:1.0:1787827405.205253:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787827405.205261:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827405.205272:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827405/real 1787827405] req@ffff8b2e8801c000 x1874672517732096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827405.205277:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517732608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827405.205326:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827405.205327:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672517732096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827405.205343:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827405.205350:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827405.205366:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827405.205366:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 411 00000100:00080000:2.0:1787827405.205374:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827405.205378:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787827405.205384:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827405.205388:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517732096:192.168.203.101@tcp:38:kworker.0 00000008:00100000:2.0:1787827405.205393:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827405.205401:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517732224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827405.205407:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517732736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827405.205448:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827405.205460:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827405/real 1787827405] req@ffff8b2e8801da40 x1874672517732224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827405.205470:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672517732224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827405.205477:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827405.205480:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827405.205482:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517732224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827405.205488:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517732352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827405.205501:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827405.205508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827405/real 1787827405] req@ffff8b2e8801fc00 x1874672517732352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827405.205517:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517732352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827405.205523:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827405.205525:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827405.205528:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517732352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827405.205541:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517732480:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827405.205554:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827405.205562:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827405/real 1787827405] req@ffff8b2e8801f480 x1874672517732480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827405.205571:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517732480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827405.205578:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827405.205580:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827405.205583:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517732480:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827405.205588:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517732608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827405.205599:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827405.205605:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827405/real 1787827405] req@ffff8b2e8801e940 x1874672517732608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827405.205614:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517732608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827405.205620:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827405.205623:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827405.205625:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517732608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827405.205630:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517732736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827405.205640:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827405.205647:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827405/real 1787827405] req@ffff8b2e8801e580 x1874672517732736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827405.205656:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517732736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827405.205684:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827405.205687:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827405.205690:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517732736:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827406.036819:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827406.228807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827407.252724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827408.084916:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827408.276783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827409.108867:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827409.300774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827410.132840:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827410.324712:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787827410.324712:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827410.324723:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827410.324739:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827410.324751:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 416 00000100:00080000:2.0:1787827410.324763:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827410.324773:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827410.324813:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d2c0 x1874672517732864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827410.324846:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827410.324849:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827410.324851:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827410.324854:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 416 00000100:00080000:2.0:1787827410.324858:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827410.324861:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827410.324871:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827410.324881:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517732864:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827410.324885:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517732992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827410.324907:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827410.324909:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827410.324910:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827410.324913:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 416 00000100:00080000:2.0:1787827410.324916:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827410.324918:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827410.324921:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827410.324932:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517733120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827410.324938:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827410.324940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827410.324942:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827410.324945:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 416 00000100:00080000:2.0:1787827410.324948:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827410.324961:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827410.324964:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787827410.324967:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827410.324976:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517733248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827410.324984:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827410.324986:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827410.324988:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827410.324991:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 416 00000100:00080000:2.0:1787827410.324994:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827410.324997:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827410.325001:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827410.325006:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff8b2e8801d2c0 x1874672517732864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827410.325012:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517733376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827410.325028:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827410.325030:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827410.325034:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827410.325035:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d2c0 x1874672517732864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827410.325043:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 416 00000100:00080000:1.0:1787827410.325050:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827410.325051:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827410.325066:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787827410.325071:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827410.325073:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517732864:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787827410.325082:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827410.325096:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517732992:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827410.325102:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517733504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827410.325140:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827410.325142:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827410.325148:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827410.325151:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827410.325156:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff8b2e8801e1c0 x1874672517732992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827410.325158:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 416 00000100:00080000:2.0:1787827410.325167:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827410.325173:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e1c0 x1874672517732992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827410.325175:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827410.325184:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827410.325186:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827410.325191:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827410.325194:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517732992:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827410.325199:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517733632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827410.325202:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517733120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827410.325229:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827410.325245:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff8b2e8801f840 x1874672517733120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827410.325253:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f840 x1874672517733120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827410.325259:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827410.325262:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827410.325264:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517733120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827410.325268:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517733248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827410.325278:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827410.325284:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff8b2e8801c780 x1874672517733248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827410.325291:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517733248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827410.325297:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827410.325299:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827410.325310:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517733248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827410.325322:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517733376:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827410.325337:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827410.325344:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff8b2e8801ed00 x1874672517733376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827410.325352:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517733376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827410.325358:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827410.325360:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827410.325362:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517733376:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827410.325382:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517733504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827410.325393:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827410.325399:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff8b2e8801f0c0 x1874672517733504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827410.325407:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517733504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827410.325412:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827410.325414:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827410.325416:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517733504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827410.325420:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517733632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827410.325430:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827410.325445:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827410/real 1787827410] req@ffff8b2e8801de00 x1874672517733632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827410.325453:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517733632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827410.325458:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827410.325460:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827410.325462:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517733632:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827411.157080:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827411.348792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827412.372806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827413.205190:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827413.396728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827414.229287:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827414.420711:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827415.252908:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827415.444790:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827415.444802:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827415.444810:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827415.444829:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827415.444830:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 421 00000100:00080000:1.0:1787827415.444860:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827415.444890:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827415.444981:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517733760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827415.445039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827415.445045:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827415.445050:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827415.445058:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 421 00000100:00080000:1.0:1787827415.445065:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827415.445073:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827415.445098:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517733760:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787827415.445111:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827415.445141:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517733888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827415.445159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827415.445164:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827415.445167:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827415.445173:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 421 00000100:00080000:1.0:1787827415.445180:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827415.445186:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827415.445194:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1787827415.445205:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827415.445212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517734016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827415.445231:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827415.445236:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827415.445239:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827415.445245:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 421 00000100:00080000:1.0:1787827415.445250:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827415.445255:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827415.445263:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827415.445275:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff8b2e880743c0 x1874672517733760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827415.445278:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517734144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827415.445303:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827415.445325:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827415.445340:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827415.445360:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 421 00000100:00080000:1.0:1787827415.445372:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827415.445382:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827415.445390:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827415.445414:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517733760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827415.445416:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517734272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827415.445479:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827415.445486:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827415.445491:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827415.445505:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 421 00000100:00080000:1.0:1787827415.445513:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827415.445514:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827415.445544:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787827415.445554:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787827415.445559:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827415.445568:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517733760:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827415.445579:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517734400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827415.445593:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827415.445598:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827415.445601:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827415.445608:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 421 00000100:00100000:2.0:1787827415.445610:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517733888:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827415.445620:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827415.445633:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:2.0:1787827415.445677:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787827415.445696:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827415.445696:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff8b2e88077840 x1874672517733888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827415.445730:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517733888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827415.445740:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517734528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827415.445745:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827415.445770:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827415.445792:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517733888:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827415.445811:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517734016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827415.445838:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827415.445847:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff8b2e880752c0 x1874672517734016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827415.445858:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517734016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827415.445865:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827415.445868:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827415.445888:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517734016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827415.445894:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517734144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827415.445912:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827415.445921:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff8b2e88074000 x1874672517734144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827415.445932:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517734144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827415.445939:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827415.445942:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827415.445945:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517734144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827415.445967:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517734272:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827415.445982:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827415.445990:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff8b2e88075e00 x1874672517734272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827415.446006:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517734272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827415.446014:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827415.446016:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827415.446018:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517734272:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827415.446024:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517734400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827415.446037:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827415.446045:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff8b2e88076d00 x1874672517734400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827415.446055:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517734400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827415.446061:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827415.446064:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827415.446066:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517734400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827415.446108:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517734528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827415.446137:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827415.446153:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827415/real 1787827415] req@ffff8b2e88074b40 x1874672517734528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827415.446166:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517734528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827415.446178:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827415.446181:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827415.446184:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517734528:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827416.276857:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827416.468810:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827417.492774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827418.324870:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827418.516754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827419.348942:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827419.540777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827420.564777:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827420.564785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827420.564789:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827420.564797:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827420.564814:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 426 00000100:00080000:1.0:1787827420.564830:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827420.564843:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827420.564899:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517734656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827420.564938:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827420.564945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827420.564949:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827420.564957:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 426 00000100:00080000:1.0:1787827420.564964:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827420.564973:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827420.564991:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827420.565000:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517734656:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827420.565015:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517734784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827420.565035:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827420.565041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827420.565072:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827420.565079:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 426 00000100:00080000:1.0:1787827420.565085:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827420.565091:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827420.565100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1787827420.565119:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827420.565123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517734912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827420.565140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827420.565146:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827420.565149:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827420.565154:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 426 00000100:00080000:1.0:1787827420.565160:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827420.565166:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827420.565174:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827420.565179:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827420/real 1787827420] req@ffff8b2e880761c0 x1874672517734656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827420.565192:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517735040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827420.565208:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827420.565210:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517734656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827420.565215:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827420.565219:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827420.565226:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 426 00000100:00080000:2.0:1787827420.565229:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827420.565232:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827420.565237:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787827420.565239:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827420.565245:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517734656:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787827420.565247:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827420.565264:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517735168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827420.565280:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827420.565281:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517734784:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827420.565287:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827420.565291:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827420.565298:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 426 00000100:00080000:1.0:1787827420.565303:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827420.565310:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827420.565318:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827420.565321:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827420.565336:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517735296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827420.565344:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827420/real 1787827420] req@ffff8b2e88076580 x1874672517734784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827420.565356:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827420.565362:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827420.565366:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827420.565369:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517734784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827420.565372:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 426 00000100:00080000:1.0:1787827420.565378:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827420.565385:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827420.565404:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827420.565415:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827420.565422:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827420.565424:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517735424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827420.565429:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517734784:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827420.565442:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517734912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827420.565484:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827420.565508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827420/real 1787827420] req@ffff8b2e880770c0 x1874672517734912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827420.565531:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517734912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827420.565542:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827420.565546:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827420.565550:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517734912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827420.565559:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517735040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827420.565586:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827420.565599:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827420/real 1787827420] req@ffff8b2e88074b40 x1874672517735040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827420.565613:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517735040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827420.565623:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827420.565627:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827420.565632:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517735040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827420.565686:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517735168:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827420.565711:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827420.565725:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827420/real 1787827420] req@ffff8b2e88076d00 x1874672517735168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827420.565740:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517735168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827420.565751:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827420.565754:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827420.565759:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517735168:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827420.565766:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517735296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827420.565783:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827420.565794:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827420/real 1787827420] req@ffff8b2e88075e00 x1874672517735296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827420.565808:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517735296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827420.565817:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827420.565820:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827420.565824:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517735296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827420.565830:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517735424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827420.565847:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827420.565859:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827420/real 1787827420] req@ffff8b2e88074000 x1874672517735424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827420.565872:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517735424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827420.565882:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827420.565885:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827420.565889:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517735424:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827421.396892:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827421.588810:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827422.612806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827423.444932:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827423.636791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827424.468822:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827424.660784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827425.492848:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827425.684800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827425.685122:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517735552:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827425.685217:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827425.685291:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827425/real 1787827425] req@ffff8b2e88038b40 x1874672517735552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827425.685341:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038b40 x1874672517735552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827425.685370:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827425.685380:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827425.685394:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517735552:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827425.685417:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517735680:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827425.685462:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827425.685486:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827425/real 1787827425] req@ffff8b2e88038780 x1874672517735680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827425.685525:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038780 x1874672517735680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827425.685541:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827425.685547:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827425.685551:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517735680:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827425.685580:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517735808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827425.685618:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827425.685632:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827425/real 1787827425] req@ffff8b2e8803b0c0 x1874672517735808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827425.685693:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517735808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827425.685704:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827425.685718:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827425.685741:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517735808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827425.685749:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517735936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827425.685777:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827425.685817:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827425/real 1787827425] req@ffff8b2e880383c0 x1874672517735936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827425.685830:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517735936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827425.685858:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827425.685861:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827425.685865:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517735936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827425.685871:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517736064:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827425.685908:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827425.685921:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827425/real 1787827425] req@ffff8b2e88039e00 x1874672517736064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827425.685935:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517736064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827425.685945:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827425.685949:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827425.685952:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517736064:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827425.685959:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517736192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827425.685977:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827425.685988:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827425/real 1787827425] req@ffff8b2e88039680 x1874672517736192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827425.686002:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517736192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827425.686064:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827425.686068:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827425.686073:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517736192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827425.686080:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517736320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827425.686105:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827425.686118:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827425/real 1787827425] req@ffff8b2e88039a40 x1874672517736320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827425.686131:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517736320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827450 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827425.686140:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827425.686143:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827425.686147:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517736320:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827426.516866:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827426.708745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827427.732752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827428.564821:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827428.756793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827429.588896:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827429.780724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827430.612840:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827430.804732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827430.804769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827430.804871:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517736448:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827430.804930:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827430.804984:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827430/real 1787827430] req@ffff8b2e88038f00 x1874672517736448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827430.805023:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517736448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827430.805042:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827430.805049:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827430.805058:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517736448:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827430.805084:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517736576:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827430.805128:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827430.805142:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827430/real 1787827430] req@ffff8b2e8803b480 x1874672517736576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827430.805151:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517736576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827430.805157:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827430.805159:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827430.805162:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517736576:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827430.805166:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517736704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827430.805185:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827430.805192:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827430/real 1787827430] req@ffff8b2e8803a1c0 x1874672517736704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827430.805200:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517736704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827430.805205:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827430.805207:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827430.805209:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517736704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827430.805213:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517736832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827430.805222:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827430.805228:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827430/real 1787827430] req@ffff8b2e8803a580 x1874672517736832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827430.805239:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517736832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827430.805244:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827430.805246:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827430.805248:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517736832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827430.805252:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517736960:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827430.805261:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827430.805267:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827430/real 1787827430] req@ffff8b2e880392c0 x1874672517736960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827430.805274:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517736960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827430.805279:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827430.805281:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827430.805283:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517736960:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827430.805293:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517737088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827430.805303:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827430.805309:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827430/real 1787827430] req@ffff8b2e8803b840 x1874672517737088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827430.805316:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517737088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827430.805325:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827430.805327:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827430.805329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517737088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827430.805333:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517737216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827430.805341:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827430.805347:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827430/real 1787827430] req@ffff8b2e88038f00 x1874672517737216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827430.805354:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517737216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827455 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827430.805359:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827430.805361:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827430.805363:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517737216:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827431.636877:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827431.828720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827431.828729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827432.852734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827432.852737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827433.684877:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827433.876733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827434.708824:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827434.900727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827435.732894:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827435.924729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787827435.924756:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827435.924762:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827435.924767:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827435.924775:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 442 00000100:00080000:2.0:1787827435.924786:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827435.924792:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827435.924830:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517737344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827435.924860:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827435.924863:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827435.924864:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827435.924868:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 442 00000100:00080000:2.0:1787827435.924871:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827435.924874:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827435.924883:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827435.924896:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517737344:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827435.924897:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517737472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827435.924913:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827435.924916:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827435.924917:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827435.924920:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 442 00000100:00080000:2.0:1787827435.924922:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827435.924925:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827435.924929:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827435.924961:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517737600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827435.924970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827435.924973:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827435.924975:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827435.924981:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 442 00000800:00000100:1.0:1787827435.924983:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827435.924985:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827435.924991:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827435.924997:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827435.925010:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517737728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827435.925019:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827435.925021:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827435.925023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827435.925026:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827435/real 1787827435] req@ffff8b2e8801c3c0 x1874672517737344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827435.925030:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 442 00000100:00080000:2.0:1787827435.925037:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827435.925044:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827435.925051:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827435.925067:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517737344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827435.925069:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517737856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827435.925086:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827435.925092:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827435.925093:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827435.925101:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517737344:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787827435.925103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827435.925109:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827435.925117:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 442 00000100:00080000:2.0:1787827435.925123:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827435.925128:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517737472:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827435.925131:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827435.925139:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827435.925153:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517737984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827435.925159:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827435.925174:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827435/real 1787827435] req@ffff8b2e8801cb40 x1874672517737472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827435.925177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827435.925186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827435.925191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827435.925196:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 442 00000100:00100000:1.0:1787827435.925200:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517737472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827435.925202:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827435.925212:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:1.0:1787827435.925214:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787827435.925217:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827435.925222:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827435.925227:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517737472:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827435.925235:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517738112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827435.925236:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517737600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827435.925259:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827435.925269:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827435/real 1787827435] req@ffff8b2e8801d680 x1874672517737600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827435.925279:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517737600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827435.925291:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827435.925293:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827435.925296:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517737600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827435.925300:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517737728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827435.925311:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827435.925318:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827435/real 1787827435] req@ffff8b2e8801de00 x1874672517737728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827435.925326:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517737728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827435.925332:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827435.925334:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827435.925337:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517737728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827435.925356:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517737856:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827435.925382:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827435.925397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827435/real 1787827435] req@ffff8b2e8801f0c0 x1874672517737856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827435.925411:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517737856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827435.925417:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827435.925420:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827435.925423:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517737856:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827435.925437:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517737984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827435.925469:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827435.925483:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827435/real 1787827435] req@ffff8b2e8801ed00 x1874672517737984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827435.925499:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517737984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827435.925509:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827435.925513:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827435.925516:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517737984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827435.925522:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517738112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827435.925539:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827435.925548:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827435/real 1787827435] req@ffff8b2e8801c780 x1874672517738112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827435.925557:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517738112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827460 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827435.925563:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827435.925566:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827435.925568:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517738112:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827436.756822:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827436.948731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827437.972710:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827438.804808:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827438.996729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827439.828788:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827440.020732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827440.852816:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827441.044726:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827441.044727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787827441.044735:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827441.044740:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827441.044747:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 447 00000100:00080000:2.0:1787827441.044756:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827441.044762:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827441.044794:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517738240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827441.044832:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827441.044835:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827441.044837:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827441.044840:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 447 00000100:00080000:2.0:1787827441.044843:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827441.044846:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827441.044856:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827441.044866:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517738240:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827441.044869:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517738368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827441.044880:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827441.044883:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827441.044884:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827441.044887:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 447 00000100:00080000:2.0:1787827441.044889:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827441.044892:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827441.044895:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:1.0:1787827441.044967:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827441.044999:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827441/real 1787827441] req@ffff8b2e8801f840 x1874672517738240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827441.045012:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f840 x1874672517738240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827441.045020:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827441.045023:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827441.045026:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517738240:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827441.045042:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517738368:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787827441.045058:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827441.045068:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827441/real 1787827441] req@ffff8b2e8801e1c0 x1874672517738368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827441.045072:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d2c0 x1874672517738496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827441.045094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827441.045095:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e1c0 x1874672517738368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827441.045100:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827441.045104:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827441.045112:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827441.045112:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 447 00000100:00080000:1.0:1787827441.045120:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787827441.045121:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787827441.045128:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517738368:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827441.045129:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827441.045137:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827441.045143:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517738496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827441.045149:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517738624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787827441.045162:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827441.045163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827441.045168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:1.0:1787827441.045172:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827441/real 1787827441] req@ffff8b2e8801d2c0 x1874672517738496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827441.045173:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827441.045185:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 447 00000100:00100000:1.0:1787827441.045189:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d2c0 x1874672517738496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827441.045192:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827441.045201:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827441.045203:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827441.045210:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827441.045216:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827441.045220:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d2c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517738496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827441.045231:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517738752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827441.045232:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517738624:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787827441.045250:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787827441.045251:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827441.045258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827441.045260:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827441.045264:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827441/real 1787827441] req@ffff8b2e8801e580 x1874672517738624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827441.045269:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 447 00000100:00080000:2.0:1787827441.045278:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827441.045300:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827441.045304:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517738624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827441.045305:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827441.045317:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827441.045321:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827441.045324:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517738880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827441.045328:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517738624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827441.045359:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517738752:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787827441.045369:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827441.045374:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:1.0:1787827441.045378:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787827441.045379:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827441.045387:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 447 00000100:00100000:1.0:1787827441.045390:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827441/real 1787827441] req@ffff8b2e8801e940 x1874672517738752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827441.045394:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827441.045403:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827441.045405:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517738752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787827441.045412:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787827441.045414:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827441.045420:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827441.045422:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517738752:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827441.045431:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517739008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827441.045433:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517738880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827441.045453:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827441.045463:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827441/real 1787827441] req@ffff8b2e8801f480 x1874672517738880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827441.045471:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517738880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827441.045479:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827441.045482:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827441.045486:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517738880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827441.045497:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517739008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787827441.045515:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827441.045527:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827441/real 1787827441] req@ffff8b2e8801fc00 x1874672517739008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827441.045537:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517739008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827441.045543:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827441.045545:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827441.045547:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517739008:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827441.876790:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827442.068718:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827443.092763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827443.924835:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827444.116731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827444.948852:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827445.140746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827445.972804:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827446.164803:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827446.164809:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827446.164813:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827446.164822:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 452 00000100:00080000:1.0:1787827446.164831:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827446.164838:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827446.164899:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517739136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827446.164926:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827446.164930:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827446.164932:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827446.164938:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 452 00000100:00080000:1.0:1787827446.164941:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827446.164945:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827446.164953:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517739136:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787827446.164954:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827446.164970:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517739264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827446.164979:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827446.164982:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827446.164984:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827446.164986:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 452 00000100:00080000:1.0:1787827446.164989:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827446.164991:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827446.164994:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827446.165002:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517739392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827446.165008:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827446.165011:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827446.165014:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827446.165014:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827446.165019:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 452 00000100:00080000:1.0:1787827446.165022:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827446.165025:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827446.165029:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827446.165036:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517739520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827446.165042:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827446.165044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827446.165046:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827446.165049:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 452 00000100:00080000:1.0:1787827446.165052:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827446.165054:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827446/real 1787827446] req@ffff8b2e880752c0 x1874672517739136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827446.165057:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827446.165063:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827446.165076:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517739648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827446.165078:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517739136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827446.165098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827446.165101:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827446.165101:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827446.165103:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827446.165106:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787827446.165108:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 452 00000100:00080000:1.0:1787827446.165112:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827446.165113:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517739136:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827446.165117:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827446.165121:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827446.165129:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517739776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827446.165136:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517739264:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827446.165137:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827446.165143:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827446.165145:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827446.165148:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 452 00000100:00080000:1.0:1787827446.165151:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827446.165153:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827446.165157:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827446.165161:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827446.165168:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517739904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827446.165173:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827446/real 1787827446] req@ffff8b2e88077840 x1874672517739264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827446.165211:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517739264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827446.165219:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827446.165223:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827446.165225:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517739264:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827446.165230:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517739392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827446.165244:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827446.165250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827446/real 1787827446] req@ffff8b2e880743c0 x1874672517739392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827446.165258:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517739392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827446.165263:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827446.165264:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827446.165266:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517739392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827446.165270:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517739520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827446.165279:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827446.165285:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827446/real 1787827446] req@ffff8b2e88074780 x1874672517739520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827446.165291:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517739520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827446.165305:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827446.165307:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827446.165309:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517739520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827446.165313:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517739648:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827446.165330:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827446.165341:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827446/real 1787827446] req@ffff8b2e88076940 x1874672517739648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827446.165372:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517739648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827446.165381:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827446.165383:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827446.165385:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517739648:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827446.165398:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517739776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827446.165412:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827446.165420:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827446/real 1787827446] req@ffff8b2e88074f00 x1874672517739776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827446.165427:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517739776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827446.165433:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827446.165435:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827446.165437:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517739776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827446.165440:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517739904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827446.165449:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827446.165454:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827446/real 1787827446] req@ffff8b2e88075680 x1874672517739904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827446.165461:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517739904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827446.165471:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827446.165473:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827446.165475:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517739904:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827446.996832:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827448.212747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827449.044823:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827449.236720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827450.068852:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827450.260728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827451.092892:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827451.284748:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827451.284748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787827451.284760:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827451.284766:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827451.284779:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 457 00000100:00080000:1.0:1787827451.284793:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827451.284803:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827451.284870:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517740032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827451.284894:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827451.284898:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827451.284900:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827451.284905:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 457 00000100:00080000:1.0:1787827451.284909:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827451.284914:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827451.284926:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827451.284931:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517740032:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827451.284939:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517740160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827451.284949:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827451.284952:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827451.284955:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827451.284958:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 457 00000100:00080000:1.0:1787827451.284961:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827451.284964:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827451.284969:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827451.284979:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517740288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827451.284988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827451.284991:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827451.284993:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827451.284996:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 457 00000100:00080000:1.0:1787827451.284999:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827451.285002:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827451.285006:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827451.285007:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827451.285015:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517740416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827451.285023:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827451.285026:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827451.285027:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827451.285031:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 457 00000100:00080000:1.0:1787827451.285034:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827451.285038:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827451.285042:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827451.285051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517740544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827451.285056:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff8b2e88077480 x1874672517740032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827451.285061:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827451.285065:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827451.285066:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827451.285070:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 457 00000100:00080000:1.0:1787827451.285073:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827451.285074:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517740032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827451.285077:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827451.285082:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827451.285086:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787827451.285090:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517740672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827451.285090:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827451.285095:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517740032:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827451.285099:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827451.285102:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827451.285103:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827451.285107:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 457 00000100:00080000:1.0:1787827451.285109:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827451.285113:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827451.285117:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827451.285119:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517740160:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827451.285126:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517740800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827451.285141:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827451.285154:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff8b2e88077c00 x1874672517740160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827451.285168:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517740160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827451.285184:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827451.285188:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827451.285191:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517740160:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827451.285198:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517740288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827451.285221:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827451.285234:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff8b2e88075a40 x1874672517740288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827451.285246:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517740288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827451.285270:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827451.285274:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827451.285277:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517740288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827451.285282:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517740416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827451.285305:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827451.285313:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff8b2e88075680 x1874672517740416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827451.285321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517740416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827451.285327:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827451.285329:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827451.285331:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517740416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827451.285336:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517740544:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827451.285346:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827451.285352:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff8b2e88074f00 x1874672517740544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827451.285360:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517740544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827451.285365:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827451.285367:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827451.285369:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517740544:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827451.285372:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517740672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827451.285381:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827451.285387:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff8b2e88076940 x1874672517740672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827451.285394:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517740672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827451.285399:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827451.285401:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827451.285403:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517740672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827451.285416:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517740800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827451.285426:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827451.285433:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827451/real 1787827451] req@ffff8b2e88074780 x1874672517740800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827451.285440:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517740800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827451.285446:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827451.285447:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827451.285449:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517740800:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827452.116871:0:5578:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827452.308765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827453.332715:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827454.164888:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827455.188818:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827456.212860:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827456.404731:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.404740:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827456.404745:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827456.404755:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 462 00000100:00080000:1.0:1787827456.404766:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.404775:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827456.404828:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517740928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827456.404854:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.404858:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827456.404861:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827456.404866:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 462 00000100:00080000:1.0:1787827456.404870:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.404875:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827456.404882:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517740928:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787827456.404885:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827456.404903:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517741056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827456.404915:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.404919:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827456.404921:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827456.404925:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 462 00000100:00080000:1.0:1787827456.404930:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.404934:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:2.0:1787827456.404939:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787827456.404940:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827456.404957:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517741184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827456.404967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.404971:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827456.404973:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827456.404976:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff8b2e880743c0 x1874672517740928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827456.404981:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 462 00000100:00080000:1.0:1787827456.404989:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.404995:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827456.404998:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517740928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827456.405002:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827456.405013:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827456.405017:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827456.405021:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517741312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827456.405023:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517740928:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827456.405037:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.405041:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827456.405044:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827456.405046:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517741056:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827456.405052:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 462 00000100:00080000:1.0:1787827456.405057:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.405063:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:2.0:1787827456.405067:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787827456.405070:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827456.405079:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff8b2e88077840 x1874672517741056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827456.405087:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517741440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827456.405111:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517741056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827456.405114:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827456.405124:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827456.405125:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827456.405127:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827456.405130:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787827456.405138:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 462 00000100:00080000:1.0:1787827456.405144:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827456.405146:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517741056:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827456.405150:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827456.405156:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517741184:192.168.203.101@tcp:8:kworker.0 00000008:00100000:1.0:1787827456.405159:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827456.405176:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517741568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827456.405179:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827456.405193:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827456.405198:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff8b2e880752c0 x1874672517741184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827456.405199:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827456.405203:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827456.405213:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 462 00000100:00100000:2.0:1787827456.405218:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517741184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827456.405219:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827456.405228:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787827456.405230:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787827456.405235:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827456.405236:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827456.405241:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517741184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827456.405248:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517741312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827456.405253:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517741696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827456.405264:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827456.405274:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff8b2e88074000 x1874672517741312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827456.405282:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517741312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827456.405292:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827456.405294:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827456.405296:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517741312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827456.405306:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517741440:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827456.405318:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827456.405324:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff8b2e88075e00 x1874672517741440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827456.405331:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517741440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827456.405337:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827456.405339:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827456.405340:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517741440:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827456.405344:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517741568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827456.405354:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827456.405359:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff8b2e88076d00 x1874672517741568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827456.405367:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517741568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827456.405374:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827456.405384:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827456.405392:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517741568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827456.405399:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517741696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827456.405420:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827456.405432:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827456/real 1787827456] req@ffff8b2e88074b40 x1874672517741696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827456.405445:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517741696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827456.405453:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827456.405457:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827456.405460:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517741696:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827459.284855:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827460.308866:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827461.332831:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827461.524718:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.524729:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827461.524737:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.524747:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 467 00000100:00080000:1.0:1787827461.524773:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827461.524781:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827461.524830:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517741824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827461.524861:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.524864:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827461.524866:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.524871:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 467 00000100:00080000:1.0:1787827461.524875:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827461.524879:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827461.524890:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827461.524890:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517741824:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827461.524905:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517741952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827461.524915:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.524918:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827461.524919:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.524922:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 467 00000100:00080000:1.0:1787827461.524925:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827461.524927:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827461.524930:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827461.524940:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517742080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827461.524946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.524949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827461.524950:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.524953:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 467 00000100:00080000:1.0:1787827461.524965:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827461.524971:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827461.524977:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827461.524991:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517742208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827461.525003:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.525008:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827461.525009:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.525013:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 467 00000100:00080000:1.0:1787827461.525017:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827461.525021:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827461.525026:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827461.525037:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517741952:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827461.525039:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517742336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827461.525050:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.525053:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827461.525055:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827461.525060:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 467 00000100:00080000:1.0:1787827461.525063:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827461.525066:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827461.525072:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827461.525072:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517742080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827461.525091:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517742464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827461.525097:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517742208:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827461.525114:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827461.525118:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827461.525120:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827461.525122:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517742336:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827461.525127:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 467 00000100:00080000:1.0:1787827461.525133:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827461.525138:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827461.525144:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827461.525146:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517742464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827461.525164:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517742592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827461.525196:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517742592:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827464.404791:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827464.596995:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787827464.597049:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827464.597073:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827464.597122:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827464.597136:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827464.597156:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:1.0:1787827464.597170:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827464] req@ffff8b2e88076580 x1874672517741952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787827464.597174:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827464.597195:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:1.0:1787827464.597220:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517741952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787827464.597220:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00080000:1.0:1787827464.597254:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827464.597261:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827464.597266:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517741952:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827464.597275:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827464] req@ffff8b2e880761c0 x1874672517742080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827464.597290:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517742080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827464.597304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827464.597312:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827464.597316:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517742080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827464.597324:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827464] req@ffff8b2e88074b40 x1874672517742208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827464.597335:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517742208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827464.597343:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827464.597346:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827464.597349:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517742208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827464.597356:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827464] req@ffff8b2e880770c0 x1874672517741824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827464.597368:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517741824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827464.597376:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827464.597379:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827464.597382:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517741824:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827464.597412:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827464] req@ffff8b2e88074000 x1874672517742592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827464.597425:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517742592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827464.597432:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827464.597436:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827464.597440:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517742592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827464.597447:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827464] req@ffff8b2e88076d00 x1874672517742336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827464.597458:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517742336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827464.597465:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827464.597469:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827464.597473:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517742336:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827464.597480:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827461/real 1787827464] req@ffff8b2e88075e00 x1874672517742464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827464.597490:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517742464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827464.597498:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827464.597501:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787827464.597504:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517742464:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827465.428812:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827466.452816:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827466.644778:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827466.644793:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827466.644806:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827466.644821:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 473 00000100:00080000:1.0:1787827466.644839:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827466.644862:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827466.644912:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517742720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827466.644961:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827466.644965:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827466.644967:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827466.644974:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 473 00000100:00080000:1.0:1787827466.644979:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827466.644986:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827466.644998:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827466.645002:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517742720:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827466.645021:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517742848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827466.645042:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827466.645047:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827466.645049:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827466.645054:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 473 00000100:00080000:1.0:1787827466.645059:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827466.645065:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827466.645071:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827466.645093:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517742976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827466.645105:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827466.645124:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827466.645128:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827466.645130:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827466.645136:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 473 00000100:00080000:1.0:1787827466.645141:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827466.645147:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827466.645153:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827466.645178:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517743104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827466.645179:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff8b2e88075e00 x1874672517742720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827466.645223:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827466.645230:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827466.645233:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827466.645244:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 473 00000100:00100000:2.0:1787827466.645245:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517742720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827466.645254:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827466.645267:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787827466.645276:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787827466.645285:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827466.645289:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827466.645299:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517742720:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827466.645308:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517743232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827466.645323:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827466.645327:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827466.645331:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827466.645338:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 473 00000100:00100000:2.0:1787827466.645344:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517742848:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827466.645348:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827466.645358:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827466.645364:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827466.645389:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517743360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827466.645404:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827466.645413:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827466.645415:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827466.645422:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827466.645430:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 473 00000100:00080000:1.0:1787827466.645436:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827466.645445:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827466.645447:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff8b2e88076d00 x1874672517742848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827466.645454:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827466.645480:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517742848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827466.645483:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517743488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827466.645563:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827466.645572:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827466.645578:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517742848:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827466.645590:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517742976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827466.645621:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827466.645636:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff8b2e88074000 x1874672517742976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827466.645689:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517742976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827466.645700:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827466.645704:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827466.645708:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517742976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827466.645716:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517743104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827466.645755:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827466.645768:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff8b2e880770c0 x1874672517743104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827466.645782:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517743104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827466.645791:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827466.645794:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827466.645798:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517743104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827466.645805:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517743232:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827466.645834:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827466.645847:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff8b2e88074b40 x1874672517743232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827466.645860:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517743232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827466.645870:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827466.645873:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827466.645877:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517743232:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827466.645898:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517743360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827466.645921:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827466.645934:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff8b2e880761c0 x1874672517743360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827466.645947:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517743360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827466.645955:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827466.645959:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827466.645962:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517743360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827466.645969:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517743488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827466.645989:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827466.646001:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827466/real 1787827466] req@ffff8b2e88076580 x1874672517743488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827466.646015:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517743488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827466.646020:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827466.646022:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827466.646024:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517743488:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827469.524850:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827470.548856:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827471.572826:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827471.764905:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517743616:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827471.765016:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827471.765091:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827471/real 1787827471] req@ffff8b2e88039a40 x1874672517743616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827471.765134:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517743616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827471.765147:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827471.765153:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827471.765157:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517743616:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827471.765188:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517743744:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827471.765212:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827471.765222:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827471/real 1787827471] req@ffff8b2e88039680 x1874672517743744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827471.765231:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517743744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827471.765242:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827471.765245:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827471.765247:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517743744:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827471.765252:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517743872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827471.765264:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827471.765270:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827471/real 1787827471] req@ffff8b2e88039e00 x1874672517743872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827471.765277:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517743872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827471.765283:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827471.765284:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827471.765286:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517743872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827471.765290:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517744000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827471.765300:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827471.765306:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827471/real 1787827471] req@ffff8b2e880383c0 x1874672517744000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827471.765314:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517744000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827471.765319:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827471.765321:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827471.765324:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517744000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827471.765328:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517744128:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827471.765338:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827471.765344:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827471/real 1787827471] req@ffff8b2e8803b0c0 x1874672517744128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827471.765351:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517744128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827471.765357:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827471.765359:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827471.765360:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517744128:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827471.765364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517744256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827471.765374:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827471.765379:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827471/real 1787827471] req@ffff8b2e88038780 x1874672517744256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827471.765386:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038780 x1874672517744256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827471.765391:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827471.765393:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827471.765395:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517744256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827471.765408:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517744384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827471.765418:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827471.765425:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827471/real 1787827471] req@ffff8b2e88038b40 x1874672517744384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827471.765432:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038b40 x1874672517744384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827496 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827471.765438:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827471.765439:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827471.765441:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517744384:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827474.644826:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827475.668818:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827476.692865:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827476.884790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827476.884920:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517744512:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827476.884975:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827476.885018:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff8b2e88038b40 x1874672517744512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827476.885049:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038b40 x1874672517744512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827476.885067:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827476.885073:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827476.885077:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517744512:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827476.885100:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517744640:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827476.885127:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827476.885143:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff8b2e88038780 x1874672517744640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827476.885159:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038780 x1874672517744640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827476.885176:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827476.885180:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827476.885183:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517744640:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827476.885189:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517744768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827476.885202:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827476.885209:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff8b2e8803b0c0 x1874672517744768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827476.885217:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517744768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827476.885222:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827476.885224:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827476.885226:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517744768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827476.885230:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517744896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827476.885239:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827476.885244:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff8b2e880383c0 x1874672517744896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827476.885251:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517744896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827476.885256:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827476.885258:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827476.885261:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517744896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827476.885265:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517745024:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827476.885274:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827476.885280:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff8b2e88039e00 x1874672517745024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827476.885287:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517745024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827476.885292:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827476.885294:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827476.885295:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517745024:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827476.885306:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517745152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827476.885315:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827476.885321:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff8b2e88039680 x1874672517745152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827476.885328:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517745152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827476.885334:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827476.885336:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827476.885338:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517745152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827476.885342:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517745280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827476.885354:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827476.885365:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827476/real 1787827476] req@ffff8b2e88039a40 x1874672517745280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827476.885378:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517745280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827501 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827476.885386:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827476.885390:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827476.885393:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517745280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827477.908761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827478.932718:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827479.764799:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827479.956775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827480.788818:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827481.812850:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827482.004813:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827482.004883:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517745408:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827482.004967:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827482.005027:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827482/real 1787827482] req@ffff8b2e88038f00 x1874672517745408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827482.005089:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517745408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827482.005123:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827482.005130:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827482.005135:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517745408:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827482.005162:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517745536:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827482.005190:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827482.005208:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827482/real 1787827482] req@ffff8b2e8803b840 x1874672517745536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827482.005224:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517745536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827482.005232:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827482.005243:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827482.005256:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517745536:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827482.005267:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517745664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827482.005302:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827482.005318:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827482/real 1787827482] req@ffff8b2e880392c0 x1874672517745664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827482.005367:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517745664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827482.005376:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827482.005381:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827482.005388:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517745664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827482.005394:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517745792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827482.005405:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827482.005411:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827482/real 1787827482] req@ffff8b2e8803a580 x1874672517745792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827482.005418:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517745792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827482.005428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827482.005431:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827482.005433:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517745792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827482.005443:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517745920:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827482.005453:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827482.005459:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827482/real 1787827482] req@ffff8b2e8803a1c0 x1874672517745920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827482.005466:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517745920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827482.005472:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827482.005474:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827482.005475:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517745920:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827482.005480:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517746048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827482.005490:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827482.005496:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827482/real 1787827482] req@ffff8b2e8803b480 x1874672517746048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827482.005503:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517746048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827482.005508:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827482.005509:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827482.005512:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517746048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827482.005516:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517746176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827482.005526:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827482.005531:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827482/real 1787827482] req@ffff8b2e88038f00 x1874672517746176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827482.005539:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517746176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827482.005543:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827482.005545:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827482.005547:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517746176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827483.028791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827484.052802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827484.884801:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827485.076814:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827485.908832:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827486.100730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827486.932842:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827487.124785:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827487.124787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827487.124799:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827487.124809:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827487.124824:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 493 00000100:00080000:1.0:1787827487.124840:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827487.124852:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827487.124901:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517746304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827487.124933:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827487.124937:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827487.124939:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827487.124943:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 493 00000100:00080000:1.0:1787827487.124947:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827487.124952:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827487.124965:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827487.124974:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517746304:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827487.124981:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517746432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827487.124994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827487.124999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827487.125001:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827487.125005:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 493 00000100:00080000:1.0:1787827487.125011:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827487.125016:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827487.125022:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827487.125035:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517746560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827487.125046:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827487.125050:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827487.125053:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827487.125060:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827487.125068:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 493 00000100:00080000:1.0:1787827487.125078:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827487.125083:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827487.125090:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827487.125104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517746688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827487.125114:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827487/real 1787827487] req@ffff8b2e880752c0 x1874672517746304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827487.125117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827487.125128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827487.125132:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827487.125138:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517746304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827487.125140:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 493 00000100:00080000:1.0:1787827487.125149:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827487.125154:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827487.125156:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787827487.125162:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787827487.125166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827487.125170:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517746304:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827487.125183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517746816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827487.125196:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827487.125201:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827487.125204:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517746432:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827487.125205:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827487.125215:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 493 00000100:00080000:1.0:1787827487.125221:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827487.125225:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827487.125231:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827487.125237:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827487.125247:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517746944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827487.125252:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827487/real 1787827487] req@ffff8b2e88077840 x1874672517746432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827487.125271:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827487.125277:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827487.125281:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827487.125281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517746432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827487.125290:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 493 00000100:00080000:1.0:1787827487.125297:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827487.125304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827487.125305:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787827487.125313:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787827487.125315:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827487.125318:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517746432:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827487.125326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517746560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827487.125332:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517747072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827487.125350:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827487.125361:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827487/real 1787827487] req@ffff8b2e880743c0 x1874672517746560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827487.125371:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517746560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827487.125378:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827487.125380:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827487.125382:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517746560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827487.125387:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517746688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827487.125397:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827487.125404:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827487/real 1787827487] req@ffff8b2e88074780 x1874672517746688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827487.125412:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517746688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827487.125417:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827487.125419:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827487.125421:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517746688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827487.125425:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517746816:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827487.125435:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827487.125441:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827487/real 1787827487] req@ffff8b2e88076940 x1874672517746816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827487.125448:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517746816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827487.125454:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827487.125456:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827487.125457:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517746816:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827487.125470:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517746944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827487.125480:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827487.125487:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827487/real 1787827487] req@ffff8b2e88074f00 x1874672517746944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827487.125494:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517746944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827487.125500:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827487.125503:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827487.125505:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517746944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827487.125508:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517747072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827487.125517:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827487.125523:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827487/real 1787827487] req@ffff8b2e88075680 x1874672517747072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827487.125530:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517747072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827487.125535:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827487.125537:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827487.125539:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517747072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827488.148747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827489.172765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827490.004852:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827490.196798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827491.028902:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827491.220763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827492.052837:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827492.244753:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827492.244755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827492.244770:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827492.244775:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.244784:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 498 00000100:00080000:1.0:1787827492.244796:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.244808:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827492.244842:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517747200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.244863:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.244866:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827492.244867:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.244870:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 498 00000100:00080000:1.0:1787827492.244873:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.244875:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827492.244884:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827492.244896:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517747328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.244908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827492.244912:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517747200:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827492.244914:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827492.244918:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.244926:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 498 00000100:00080000:1.0:1787827492.244930:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.244935:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827492.244953:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827492.244969:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517747456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.244976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.244979:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827492.244980:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.244983:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 498 00000100:00080000:1.0:1787827492.244989:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.244992:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827492.244997:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827492.245005:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827492.245010:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517747584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.245018:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.245020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827492.245021:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.245024:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 498 00000100:00080000:1.0:1787827492.245030:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.245033:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827492.245037:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827492.245045:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517747712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.245051:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.245053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827492.245055:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.245058:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 498 00000100:00080000:1.0:1787827492.245063:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.245066:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827492.245070:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827492.245078:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827492/real 1787827492] req@ffff8b2e88075a40 x1874672517747200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827492.245082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517747840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827492.245120:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827492.245123:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827492.245125:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827492.245132:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 498 00000100:00100000:2.0:1787827492.245132:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517747200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827492.245134:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827492.245143:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827492.245150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827492.245169:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517747968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827492.245169:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827492.245182:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827492.245187:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517747200:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827492.245213:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517747328:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827492.245236:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827492.245247:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827492/real 1787827492] req@ffff8b2e88077c00 x1874672517747328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827492.245256:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517747328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827492.245262:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827492.245265:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827492.245267:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517747328:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827492.245271:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517747456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827492.245284:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827492.245290:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827492/real 1787827492] req@ffff8b2e88077480 x1874672517747456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827492.245298:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517747456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827492.245304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827492.245306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827492.245307:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517747456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827492.245312:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517747584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827492.245322:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827492.245328:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827492/real 1787827492] req@ffff8b2e88075680 x1874672517747584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827492.245336:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517747584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827492.245341:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827492.245343:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827492.245345:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517747584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827492.245349:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517747712:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827492.245362:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827492.245373:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827492/real 1787827492] req@ffff8b2e88074f00 x1874672517747712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827492.245385:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517747712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827492.245394:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827492.245398:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827492.245401:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517747712:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827492.245408:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517747840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827492.245431:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827492.245443:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827492/real 1787827492] req@ffff8b2e88076940 x1874672517747840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827492.245457:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517747840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827492.245466:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827492.245470:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827492.245473:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517747840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827492.245480:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517747968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827492.245501:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827492.245513:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827492/real 1787827492] req@ffff8b2e88074780 x1874672517747968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827492.245535:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517747968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827492.245558:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827492.245563:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827492.245567:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517747968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827493.269434:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827495.124842:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827496.148877:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827497.172791:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827497.364820:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.364829:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827497.364836:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827497.364840:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827497.364847:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 503 00000100:00080000:1.0:1787827497.364862:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.364871:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827497.364911:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517748096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827497.364959:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.364963:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827497.364965:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827497.364970:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 503 00000100:00080000:1.0:1787827497.364975:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.364979:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827497.364990:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827497.365002:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517748224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827497.365010:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517748096:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827497.365014:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.365018:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827497.365019:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827497.365023:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 503 00000100:00080000:1.0:1787827497.365025:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.365028:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827497.365032:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827497.365040:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517748352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827497.365046:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.365049:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827497.365050:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827497.365053:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 503 00000100:00080000:1.0:1787827497.365055:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.365057:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827497.365061:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827497.365069:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517748480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827497.365073:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827497.365086:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.365090:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827497.365092:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827497.365097:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 503 00000100:00080000:1.0:1787827497.365102:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.365106:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827497.365112:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827497.365114:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff8b2e880743c0 x1874672517748096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827497.365130:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517748608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827497.365132:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517748096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827497.365156:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827497.365158:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827497.365161:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827497.365164:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787827497.365165:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827497.365171:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517748096:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827497.365173:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 503 00000100:00080000:1.0:1787827497.365180:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.365191:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827497.365194:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517748224:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787827497.365198:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827497.365213:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517748736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827497.365221:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827497.365225:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827497.365229:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827497.365231:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827497.365234:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff8b2e88077840 x1874672517748224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827497.365235:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 503 00000100:00080000:1.0:1787827497.365241:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827497.365249:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827497.365255:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517748224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827497.365258:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787827497.365265:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827497.365269:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827497.365271:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517748224:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827497.365278:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517748864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827497.365279:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517748352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827497.365325:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827497.365341:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff8b2e880752c0 x1874672517748352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827497.365356:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517748352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827497.365366:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827497.365370:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827497.365374:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517748352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827497.365380:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517748480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827497.365397:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827497.365406:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff8b2e88076580 x1874672517748480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827497.365415:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517748480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827497.365421:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827497.365423:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827497.365426:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517748480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827497.365447:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517748608:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827497.365457:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827497.365463:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff8b2e880761c0 x1874672517748608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827497.365470:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517748608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827497.365475:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827497.365477:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827497.365479:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517748608:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827497.365491:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517748736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827497.365501:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827497.365508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff8b2e88074b40 x1874672517748736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827497.365530:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517748736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827497.365535:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827497.365537:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827497.365539:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517748736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827497.365560:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517748864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827497.365578:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827497.365587:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827497/real 1787827497] req@ffff8b2e880770c0 x1874672517748864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827497.365597:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517748864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827497.365604:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827497.365606:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827497.365609:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517748864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827498.388794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827499.412781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827500.244832:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827500.436729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827501.269009:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827501.460767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827502.293739:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827502.484775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827502.484779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827502.484910:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517748992:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827502.484998:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827502.485046:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827502] req@ffff8b2e880412c0 x1874672517748992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827502.485087:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672517748992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827502.485113:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827502.485121:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827502.485126:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517748992:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827502.485149:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517749120:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827502.485174:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827502.485188:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827502] req@ffff8b2e88042580 x1874672517749120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827502.485212:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672517749120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827502.485221:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827502.485225:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827502.485229:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517749120:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827502.485235:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517749248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827502.485257:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827502.485279:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827502] req@ffff8b2e88040780 x1874672517749248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827502.485292:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517749248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827502.485307:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827502.485311:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827502.485315:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517749248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827502.485321:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517749376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827502.485342:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827502.485353:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827502] req@ffff8b2e88043840 x1874672517749376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827502.485365:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517749376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827502.485374:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827502.485377:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827502.485381:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517749376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827502.485388:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517749504:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827502.485408:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827502.485420:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827502] req@ffff8b2e88042d00 x1874672517749504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827502.485432:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517749504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827502.485442:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827502.485445:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827502.485450:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517749504:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827502.485456:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517749632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827502.485496:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827502.485508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827502] req@ffff8b2e880421c0 x1874672517749632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827502.485521:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517749632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827502.485529:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827502.485532:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827502.485536:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517749632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827502.485542:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517749760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827502.485563:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827502.485574:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827502/real 1787827502] req@ffff8b2e88040b40 x1874672517749760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827502.485586:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517749760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827502.485594:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827502.485598:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827502.485601:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517749760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827503.508750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827503.508776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827504.532725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827504.532728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827505.364772:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827505.556734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827506.388791:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827506.580790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827507.412932:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827507.604772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827507.604945:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517749888:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827507.605023:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827507.605070:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff8b2e88041e00 x1874672517749888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827507.605116:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517749888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827507.605149:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827507.605159:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827507.605167:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517749888:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827507.605204:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517750016:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827507.605237:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827507.605253:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff8b2e880403c0 x1874672517750016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827507.605318:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672517750016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827507.605332:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827507.605337:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827507.605340:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517750016:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827507.605348:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517750144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827507.605370:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827507.605379:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff8b2e880430c0 x1874672517750144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827507.605388:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517750144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827507.605394:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827507.605396:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827507.605398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517750144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827507.605402:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517750272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827507.605413:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827507.605418:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff8b2e88040000 x1874672517750272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827507.605426:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517750272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827507.605431:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827507.605433:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827507.605435:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517750272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827507.605455:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517750400:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827507.605471:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827507.605478:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff8b2e88040f00 x1874672517750400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827507.605491:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517750400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827507.605496:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827507.605499:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827507.605501:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517750400:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827507.605514:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517750528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827507.605525:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827507.605531:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff8b2e88042940 x1874672517750528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827507.605538:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517750528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827507.605544:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827507.605546:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827507.605548:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517750528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827507.605552:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517750656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827507.605561:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827507.605566:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827507/real 1787827507] req@ffff8b2e88043480 x1874672517750656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827507.605583:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517750656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827507.605597:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827507.605601:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827507.605604:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517750656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827508.628755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827509.652816:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827510.484880:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827510.676742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827511.508771:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827511.700742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827512.532789:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827512.724723:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827512.724854:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517750784:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827512.724913:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827512.724951:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827512/real 1787827512] req@ffff8b2e88043c00 x1874672517750784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827512.724976:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043c00 x1874672517750784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827512.724990:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827512.724999:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827512.725004:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517750784:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827512.725029:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517750912:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827512.725054:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827512.725067:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827512/real 1787827512] req@ffff8b2e88041a40 x1874672517750912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827512.725080:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517750912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827512.725091:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827512.725101:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827512.725108:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517750912:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827512.725118:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517751040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827512.725143:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827512.725156:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827512/real 1787827512] req@ffff8b2e88043480 x1874672517751040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827512.725166:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517751040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827512.725172:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827512.725175:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827512.725177:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517751040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827512.725181:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517751168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827512.725202:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827512.725209:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827512/real 1787827512] req@ffff8b2e88042940 x1874672517751168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827512.725216:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517751168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827512.725222:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827512.725224:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827512.725226:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517751168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827512.725236:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517751296:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827512.725247:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827512.725254:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827512/real 1787827512] req@ffff8b2e88040f00 x1874672517751296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827512.725261:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517751296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827512.725266:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827512.725268:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827512.725270:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517751296:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827512.725273:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517751424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827512.725282:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827512.725288:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827512/real 1787827512] req@ffff8b2e88040000 x1874672517751424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827512.725296:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517751424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827512.725304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827512.725307:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827512.725311:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517751424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827512.725318:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517751552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827512.725347:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827512.725360:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827512/real 1787827512] req@ffff8b2e880430c0 x1874672517751552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827512.725373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517751552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827537 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827512.725381:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827512.725385:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827512.725387:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517751552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827513.748737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827514.772727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827515.604837:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827515.796722:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827516.628857:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827516.820759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827517.652827:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827517.844746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827517.844933:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517751680:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827517.844998:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827517.845040:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827517/real 1787827517] req@ffff8b2e880403c0 x1874672517751680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827517.845079:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672517751680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827517.845095:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827517.845102:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827517.845112:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517751680:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827517.845137:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517751808:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827517.845174:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827517.845192:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827517/real 1787827517] req@ffff8b2e88041e00 x1874672517751808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827517.845208:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517751808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827517.845222:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827517.845229:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827517.845234:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517751808:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827517.845250:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517751936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827517.845273:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827517.845288:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827517/real 1787827517] req@ffff8b2e88040b40 x1874672517751936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827517.845320:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517751936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827517.845346:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827517.845360:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827517.845381:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517751936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827517.845401:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517752064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827517.845424:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827517.845437:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827517/real 1787827517] req@ffff8b2e880421c0 x1874672517752064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827517.845450:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517752064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827517.845458:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827517.845462:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827517.845465:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517752064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827517.845471:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517752192:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827517.845488:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827517.845498:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827517/real 1787827517] req@ffff8b2e88042d00 x1874672517752192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827517.845525:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517752192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827517.845537:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827517.845547:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827517.845562:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517752192:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827517.845600:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517752320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827517.845632:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827517.845690:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827517/real 1787827517] req@ffff8b2e88043840 x1874672517752320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827517.845706:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517752320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827517.845713:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827517.845716:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827517.845719:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517752320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827517.845730:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517752448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827517.845746:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827517.845754:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827517/real 1787827517] req@ffff8b2e88040780 x1874672517752448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827517.845761:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517752448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827542 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827517.845767:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827517.845769:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827517.845771:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517752448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827518.868728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827519.892739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827520.724895:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827520.916871:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827521.748804:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827521.940744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827522.772894:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827522.964787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827522.964905:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517752576:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827522.965013:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827522.965077:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff8b2e88042580 x1874672517752576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827522.965145:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672517752576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827522.965169:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827522.965178:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827522.965187:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517752576:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827522.965221:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517752704:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827522.965247:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827522.965258:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff8b2e880412c0 x1874672517752704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827522.965267:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672517752704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827522.965273:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827522.965276:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827522.965278:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517752704:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827522.965282:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517752832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827522.965295:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827522.965301:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff8b2e88040780 x1874672517752832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827522.965309:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517752832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827522.965315:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827522.965317:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827522.965319:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517752832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827522.965323:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517752960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827522.965349:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827522.965359:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff8b2e88043840 x1874672517752960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827522.965367:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517752960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827522.965373:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827522.965376:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827522.965380:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517752960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827522.965387:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517753088:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827522.965410:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827522.965422:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff8b2e88042d00 x1874672517753088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827522.965436:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517753088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827522.965442:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827522.965444:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827522.965447:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517753088:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827522.965451:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517753216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827522.965467:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827522.965474:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff8b2e880421c0 x1874672517753216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827522.965482:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517753216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827522.965488:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827522.965490:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827522.965492:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517753216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827522.965506:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517753344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827522.965517:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827522.965523:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827522/real 1787827522] req@ffff8b2e88040b40 x1874672517753344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827522.965530:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517753344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827547 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827522.965536:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827522.965538:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827522.965540:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517753344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827523.988760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827525.012760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827525.844856:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827526.036749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827526.868865:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827527.060746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827527.892882:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827528.084753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827528.084797:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827528.084810:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827528.084816:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827528.084827:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 534 00000100:00080000:1.0:1787827528.084840:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827528.084850:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827528.084887:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517753472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827528.084915:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827528.084919:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827528.084921:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827528.084925:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 534 00000100:00080000:1.0:1787827528.084929:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827528.084934:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827528.084944:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827528.084952:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517753472:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827528.084957:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517753600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827528.084969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827528.084974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827528.084976:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827528.084981:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 534 00000100:00080000:1.0:1787827528.084986:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827528.084990:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827528.084996:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827528.085009:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517753728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827528.085026:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827528.085029:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827528.085031:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827528.085036:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 534 00000100:00080000:1.0:1787827528.085041:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827528.085050:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517753600:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827528.085051:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827528.085063:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827528.085081:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517753856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827528.085085:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517753728:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827528.085108:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827528.085113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827528.085117:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827528.085122:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 534 00000100:00080000:1.0:1787827528.085126:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827528.085131:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827528.085148:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827528.085149:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517753856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827528.085169:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517753984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827528.085190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827528.085194:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827528.085196:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827528.085202:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 534 00000100:00080000:1.0:1787827528.085207:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827528.085210:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517753984:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827528.085215:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827528.085224:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827528.085250:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517754112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827528.085266:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827528.085270:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827528.085272:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827528.085277:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 534 00000100:00080000:1.0:1787827528.085282:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827528.085282:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517754112:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787827528.085292:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827528.085310:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827528.085330:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517754240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827528.085377:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517754240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827529.108735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827530.132770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827530.964806:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827531.156719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:1.0:1787827531.156947:0:2336:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787827531.156987:0:2336:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827531.157004:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827531.157050:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827531.157060:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827531.157068:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827531.157076:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827531.157085:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827531.157092:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:2.0:1787827531.157523:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827528/real 1787827531] req@ffff8b2e88076580 x1874672517754240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827531.157550:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517754240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827531.157564:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827531.157590:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827531.157595:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517754240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827531.157604:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827528/real 1787827531] req@ffff8b2e880761c0 x1874672517754112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827531.157618:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517754112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827531.157627:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827531.157631:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827531.157634:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517754112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827531.157641:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827528/real 1787827531] req@ffff8b2e88074b40 x1874672517753984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827531.157685:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517753984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827531.157694:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827531.157697:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827531.157701:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517753984:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827531.157708:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827528/real 1787827531] req@ffff8b2e880770c0 x1874672517753856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827531.157721:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517753856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827531.157729:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827531.157734:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827531.157738:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517753856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827531.157744:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827528/real 1787827531] req@ffff8b2e88076d00 x1874672517753600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827531.157757:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517753600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827531.157765:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827531.157768:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827531.157772:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517753600:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827531.157779:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827528/real 1787827531] req@ffff8b2e88075e00 x1874672517753728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827531.157792:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517753728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827531.157801:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827531.157805:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827531.157808:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517753728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827531.157815:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827528/real 1787827531] req@ffff8b2e88074000 x1874672517753472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827531.157837:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517753472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827531.157848:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827531.157853:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827531.157858:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517753472:192.168.203.101@tcp:250:kworker.0 00010000:00010000:1.0:1787827531.988966:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827532.180807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827532.180886:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827533.012835:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827533.204833:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787827533.204835:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827533.204847:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827533.204854:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827533.204870:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 539 00000100:00080000:1.0:1787827533.204884:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827533.204895:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827533.204943:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517754368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827533.204976:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827533.204980:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827533.204982:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827533.204986:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 539 00000100:00080000:1.0:1787827533.204989:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827533.204992:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827533.205005:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827533.205023:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517754496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827533.205036:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517754368:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827533.205040:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827533.205049:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827533.205051:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827533.205082:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 539 00000100:00080000:1.0:1787827533.205089:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827533.205095:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827533.205103:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827533.205136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517754624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827533.205147:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827533.205151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827533.205153:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827533.205156:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 539 00000100:00080000:1.0:1787827533.205159:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827533.205162:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827533.205166:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827533.205171:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827533.205179:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517754752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827533.205188:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827533.205191:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827533.205193:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827533.205196:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 539 00000100:00080000:1.0:1787827533.205199:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827533.205203:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827533.205208:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827533.205219:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517754880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827533.205228:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827533.205232:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827533/real 1787827533] req@ffff8b2e880752c0 x1874672517754368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827533.205233:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827533.205237:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827533.205246:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 539 00000100:00080000:1.0:1787827533.205252:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827533.205257:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827533.205278:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517754368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827533.205280:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827533.205302:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517755008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827533.205304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827533.205314:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787827533.205321:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827533.205322:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517754368:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827533.205328:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827533.205332:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827533.205338:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 539 00000100:00080000:1.0:1787827533.205342:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827533.205346:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827533.205352:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827533.205363:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517754496:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787827533.205366:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517755136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827533.205398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827533.205417:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827533/real 1787827533] req@ffff8b2e88077840 x1874672517754496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827533.205430:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517754496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827533.205439:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827533.205443:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827533.205446:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517754496:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827533.205453:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517754624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827533.205477:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827533.205495:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827533/real 1787827533] req@ffff8b2e880743c0 x1874672517754624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827533.205504:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517754624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827533.205513:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827533.205516:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827533.205520:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517754624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827533.205527:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517754752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827533.205547:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827533.205558:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827533/real 1787827533] req@ffff8b2e88074780 x1874672517754752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827533.205571:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517754752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827533.205579:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827533.205583:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827533.205586:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517754752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827533.205592:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517754880:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827533.205613:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827533.205624:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827533/real 1787827533] req@ffff8b2e88076940 x1874672517754880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827533.205635:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517754880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827533.205643:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827533.205676:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827533.205680:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517754880:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827533.205687:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517755008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827533.205709:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827533.205721:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827533/real 1787827533] req@ffff8b2e88074f00 x1874672517755008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827533.205736:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517755008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827533.205745:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827533.205748:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827533.205752:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517755008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827533.205775:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517755136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827533.205800:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827533.205813:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827533/real 1787827533] req@ffff8b2e88075680 x1874672517755136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827533.205822:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517755136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827533.205829:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827533.205832:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827533.205835:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517755136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827534.228810:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827534.228902:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827535.252736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827535.252813:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827536.084875:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827536.276877:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827537.108839:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827537.300768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827537.300840:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827538.132942:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827538.324721:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827538.324730:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827538.324734:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827538.324747:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 544 00000100:00080000:1.0:1787827538.324770:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827538.324777:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827538.324815:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517755264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827538.324843:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827538.324846:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827538.324860:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827538.324863:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827538.324867:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 544 00000100:00080000:1.0:1787827538.324874:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827538.324879:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827538.324890:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827538.324907:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517755392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827538.324919:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827538.324935:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827538.324937:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827538.324940:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517755264:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827538.324942:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 544 00000100:00080000:1.0:1787827538.324948:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827538.324958:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827538.324964:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827538.324971:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517755520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827538.324977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827538.324980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827538.324981:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827538.324984:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 544 00000100:00080000:1.0:1787827538.324986:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827538.324988:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827538.324991:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827538.325013:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517755648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827538.325020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827538.325022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827538.325023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827538.325026:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 544 00000100:00080000:1.0:1787827538.325029:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827538.325031:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827538.325034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827538.325041:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517755776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827538.325047:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827538.325050:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827538.325051:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827538.325055:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 544 00000100:00080000:1.0:1787827538.325057:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827538.325060:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827538.325063:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827538.325072:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517755904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827538.325078:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827538.325079:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827538.325081:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827538.325084:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 544 00000100:00080000:1.0:1787827538.325095:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827538.325099:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827538.325115:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827538.325123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517756032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787827538.325135:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827538.325253:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff8b2e88077480 x1874672517755264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827538.325310:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517755264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827538.325327:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827538.325332:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827538.325338:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517755264:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827538.325373:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517755392:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827538.325425:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827538.325442:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff8b2e88077c00 x1874672517755392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827538.325457:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517755392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827538.325468:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827538.325472:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827538.325494:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517755392:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827538.325502:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517755520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827538.325526:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827538.325538:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff8b2e88075a40 x1874672517755520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827538.325562:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517755520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827538.325572:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827538.325575:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827538.325579:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517755520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827538.325586:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517755648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827538.325605:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827538.325616:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff8b2e88075680 x1874672517755648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827538.325630:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517755648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827538.325639:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827538.325643:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827538.325647:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517755648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827538.325672:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517755776:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827538.325692:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827538.325703:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff8b2e88074f00 x1874672517755776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827538.325769:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517755776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827538.325789:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827538.325793:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827538.325797:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517755776:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827538.325805:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517755904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827538.325830:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827538.325844:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff8b2e88076940 x1874672517755904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827538.325867:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517755904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827538.325877:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827538.325881:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827538.325885:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517755904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827538.325893:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517756032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827538.325911:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827538.325942:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827538/real 1787827538] req@ffff8b2e88074780 x1874672517756032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827538.325955:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517756032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827538.325964:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827538.325967:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827538.325972:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517756032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827539.348757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827540.372778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827541.204880:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827541.396820:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827542.228949:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827542.420770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827543.252899:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827543.444766:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827543.444786:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827543.444790:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827543.444798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827543.444805:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 549 00000100:00080000:1.0:1787827543.444827:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827543.444841:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827543.444880:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517756160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827543.444919:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827543.444923:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827543.444926:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827543.444929:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 549 00000100:00080000:1.0:1787827543.444936:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827543.444944:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827543.444953:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827543.444974:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517756288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827543.444983:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827543.444986:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827543.444987:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827543.444989:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 549 00000100:00080000:1.0:1787827543.444992:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827543.444994:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827543.444997:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827543.445005:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517756416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827543.445011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827543.445014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827543.445015:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827543.445024:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 549 00000100:00080000:1.0:1787827543.445028:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827543.445030:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827543.445034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827543.445044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517756544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827543.445050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827543.445052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827543.445054:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827543.445057:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 549 00000100:00080000:1.0:1787827543.445060:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827543.445062:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827543.445068:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827543.445081:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517756672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827543.445091:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827543.445098:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827543.445101:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827543.445107:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 549 00000100:00080000:1.0:1787827543.445113:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827543.445119:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827543.445125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827543.445138:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517756800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827543.445148:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827543.445152:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827543.445157:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827543.445161:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 549 00000100:00080000:1.0:1787827543.445164:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827543.445166:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827543.445169:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827543.445181:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517756928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827544.468846:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827545.492799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827546.324902:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827546.516815:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827547.348855:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827547.540779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827548.372800:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827548.564763:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827548.564770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827548.564775:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827548.564792:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827548.564812:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 554 00000100:00080000:1.0:1787827548.564825:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827548.564833:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827548.564872:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517757056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827548.564905:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827548.564909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827548.564911:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827548.564915:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 554 00000100:00080000:1.0:1787827548.564918:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827548.564923:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827548.564933:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827548.564950:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517757184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827548.564987:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827548.564990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827548.564992:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827548.564997:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 554 00000100:00080000:1.0:1787827548.564999:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827548.565002:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827548.565006:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827548.565016:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517757312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827548.565023:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827548.565026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827548.565044:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827548.565047:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 554 00000100:00080000:1.0:1787827548.565056:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827548.565060:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827548.565065:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827548.565082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517757440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827548.565092:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827548.565111:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827548.565114:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827548.565119:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 554 00000100:00080000:1.0:1787827548.565122:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827548.565128:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827548.565136:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827548.565150:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517757568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827548.565231:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827548.565238:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827548.565239:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827548.565247:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 554 00000100:00080000:1.0:1787827548.565253:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827548.565261:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827548.565272:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827548.565292:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517757696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827548.565313:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827548.565318:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827548.565321:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827548.565336:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 554 00000100:00080000:1.0:1787827548.565340:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827548.565342:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827548.565345:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827548.565355:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517757824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827549.588767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827550.612781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827551.444935:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827551.636790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827552.468850:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827552.660783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827553.492840:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827553.684723:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827553.684733:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827553.684737:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827553.684747:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 560 00000100:00080000:1.0:1787827553.684758:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827553.684765:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827553.684808:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517757952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827553.684815:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827553.684838:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827553.684842:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827553.684844:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827553.684849:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 560 00000100:00080000:1.0:1787827553.684853:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827553.684859:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827553.684868:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827553.684881:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517758080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827553.684892:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827553.684895:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827553.684896:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827553.684901:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 560 00000100:00080000:1.0:1787827553.684903:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827553.684905:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827553.684909:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827553.684917:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517758208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827553.684923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827553.684926:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827553.684927:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827553.684930:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 560 00000100:00080000:1.0:1787827553.684932:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827553.684934:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827553.684938:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827553.684950:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517758336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827553.684958:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827553.684961:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827553.684962:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827553.684965:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 560 00000100:00080000:1.0:1787827553.684967:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827553.684970:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827553.684973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827553.684982:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517758464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827553.685004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827553.685007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827553.685009:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827553.685013:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 560 00000100:00080000:1.0:1787827553.685016:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827553.685023:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827553.685029:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827553.685044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517758592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827553.685061:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827553.685066:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827553.685068:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827553.685077:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 560 00000100:00080000:1.0:1787827553.685084:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827553.685089:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827553.685096:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827553.685111:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517758720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827554.708795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827555.732760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827556.564942:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827556.756847:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827557.588838:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827557.780752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827558.612916:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827558.804759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827558.804969:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517758848:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827558.805073:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827558.805155:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff8b2e88039a40 x1874672517758848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827558.805185:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517758848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.805221:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.805228:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827558.805241:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517758848:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827558.805276:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517758976:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827558.805299:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827558.805309:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff8b2e88039680 x1874672517758976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827558.805317:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517758976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.805324:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.805327:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827558.805329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517758976:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827558.805354:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517759104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827558.805376:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827558.805397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff8b2e88039e00 x1874672517759104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827558.805410:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517759104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.805416:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.805419:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827558.805421:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517759104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827558.805427:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517759232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827558.805442:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827558.805450:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff8b2e880383c0 x1874672517759232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827558.805459:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517759232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.805465:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.805467:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827558.805469:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517759232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827558.805473:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517759360:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827558.805483:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827558.805489:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff8b2e8803b0c0 x1874672517759360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827558.805504:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517759360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.805510:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.805513:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827558.805515:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517759360:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827558.805519:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517759488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827558.805530:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827558.805536:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff8b2e88038780 x1874672517759488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827558.805544:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038780 x1874672517759488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.805554:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.805556:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827558.805558:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517759488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827558.805562:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517759616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827558.805572:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827558.805577:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827558/real 1787827558] req@ffff8b2e88038b40 x1874672517759616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827558.805584:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038b40 x1874672517759616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827583 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827558.805590:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827558.805592:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827558.805594:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517759616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827559.828757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827560.852740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827561.684844:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827561.876772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827562.708907:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827562.900778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827563.732814:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827563.924750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827563.924767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827563.924906:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517759744:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827563.925004:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827563.925101:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff8b2e88038b40 x1874672517759744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.925137:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038b40 x1874672517759744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.925152:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.925157:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827563.925163:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517759744:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827563.925189:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517759872:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827563.925216:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827563.925229:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff8b2e88038780 x1874672517759872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.925245:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038780 x1874672517759872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.925254:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.925258:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827563.925262:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517759872:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827563.925269:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517760000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827563.925289:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827563.925300:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff8b2e8803b0c0 x1874672517760000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.925313:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517760000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.925323:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.925326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827563.925330:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517760000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827563.925337:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517760128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827563.925355:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827563.925366:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff8b2e880383c0 x1874672517760128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.925379:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517760128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.925388:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.925392:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827563.925395:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517760128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827563.925402:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517760256:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827563.925421:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827563.925432:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff8b2e88039e00 x1874672517760256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.925445:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517760256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.925455:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.925458:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827563.925461:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517760256:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827563.925467:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517760384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827563.925485:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827563.925496:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff8b2e88039680 x1874672517760384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.925509:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517760384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.925518:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.925522:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827563.925526:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517760384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827563.925533:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517760512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827563.925551:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827563.925561:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827563/real 1787827563] req@ffff8b2e88039a40 x1874672517760512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827563.925575:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517760512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827588 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827563.925584:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827563.925587:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827563.925591:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517760512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827564.948792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827564.948792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827565.972818:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827565.972818:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827566.804871:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827566.996814:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827567.828853:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827568.852888:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827569.044838:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827569.044956:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517760640:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827569.045065:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827569.045121:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827569/real 1787827569] req@ffff8b2e88038f00 x1874672517760640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827569.045147:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517760640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827569.045179:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827569.045186:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827569.045195:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517760640:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827569.045221:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517760768:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827569.045246:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827569.045256:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827569/real 1787827569] req@ffff8b2e8803b480 x1874672517760768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827569.045265:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517760768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827569.045286:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827569.045288:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827569.045291:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517760768:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827569.045295:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517760896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827569.045309:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827569.045316:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827569/real 1787827569] req@ffff8b2e8803a1c0 x1874672517760896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827569.045323:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517760896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827569.045329:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827569.045331:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827569.045333:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517760896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827569.045338:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517761024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827569.045347:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827569.045381:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827569/real 1787827569] req@ffff8b2e8803a580 x1874672517761024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827569.045391:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517761024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827569.045398:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827569.045400:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827569.045402:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517761024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827569.045406:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517761152:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827569.045419:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827569.045426:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827569/real 1787827569] req@ffff8b2e880392c0 x1874672517761152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827569.045435:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517761152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827569.045453:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827569.045456:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827569.045458:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517761152:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827569.045462:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517761280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827569.045474:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827569.045482:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827569/real 1787827569] req@ffff8b2e8803b840 x1874672517761280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827569.045491:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517761280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827569.045498:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827569.045500:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827569.045502:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517761280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827569.045506:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517761408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827569.045518:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827569.045525:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827569/real 1787827569] req@ffff8b2e88039a40 x1874672517761408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827569.045533:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517761408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827569.045554:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827569.045556:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827569.045558:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517761408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827570.068767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827571.092802:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827571.924911:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827572.116734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827572.948872:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827573.140752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827573.972880:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827574.164812:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827574.164935:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517761536:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827574.165078:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827574.165204:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827574/real 1787827574] req@ffff8b2e88039680 x1874672517761536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827574.165287:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517761536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827574.165303:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827574.165309:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827574.165314:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517761536:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827574.165344:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517761664:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827574.165372:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827574.165386:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827574/real 1787827574] req@ffff8b2e88039e00 x1874672517761664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827574.165401:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517761664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827574.165411:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827574.165434:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827574.165439:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517761664:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827574.165446:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517761792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827574.165467:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827574.165479:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827574/real 1787827574] req@ffff8b2e880383c0 x1874672517761792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827574.165493:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517761792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827574.165510:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827574.165514:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827574.165518:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517761792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827574.165525:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517761920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827574.165543:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827574.165561:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827574/real 1787827574] req@ffff8b2e8803b0c0 x1874672517761920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827574.165576:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517761920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827574.165585:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827574.165590:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827574.165593:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517761920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827574.165601:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517762048:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827574.165621:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827574.165632:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827574/real 1787827574] req@ffff8b2e88038780 x1874672517762048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827574.165669:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038780 x1874672517762048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827574.165681:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827574.165685:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827574.165689:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517762048:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827574.165697:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517762176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827574.165717:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827574.165729:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827574/real 1787827574] req@ffff8b2e88038b40 x1874672517762176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827574.165742:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038b40 x1874672517762176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827574.165751:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827574.165755:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827574.165759:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517762176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827574.165766:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517762304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827574.165785:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827574.165795:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827574/real 1787827574] req@ffff8b2e88039a40 x1874672517762304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827574.165808:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517762304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827574.165817:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827574.165821:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827574.165825:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517762304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827575.188768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827576.212758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827577.044834:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827578.068879:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827579.092841:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827579.284727:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827579.284735:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827579.284739:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827579.284751:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 585 00000100:00080000:1.0:1787827579.284764:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827579.284776:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827579.284814:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517762432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827579.284831:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827579.284834:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827579.284836:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827579.284840:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 585 00000100:00080000:1.0:1787827579.284843:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827579.284846:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827579.284853:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827579.284854:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517762432:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827579.284865:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517762560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827579.284874:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827579.284877:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827579.284878:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827579.284880:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 585 00000100:00080000:1.0:1787827579.284882:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827579.284884:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827579.284887:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827579.284896:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517762688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827579.284902:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827579.284905:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827579.284906:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827579.284908:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 585 00000100:00080000:1.0:1787827579.284910:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827579.284914:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:2.0:1787827579.284915:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787827579.284919:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827579.284930:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517762816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827579.284936:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827579.284938:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827579.284957:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827579.284960:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 585 00000100:00080000:1.0:1787827579.284963:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827579.284969:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827579.284970:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff8b2e88075a40 x1874672517762432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827579.284973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827579.284988:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517762944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827579.284992:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517762432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827579.285000:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827579.285007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827579.285008:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827579.285010:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827579.285014:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 585 00000100:00080000:2.0:1787827579.285017:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787827579.285019:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827579.285025:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517762432:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827579.285026:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827579.285032:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827579.285041:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517763072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827579.285049:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827579.285052:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517762560:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787827579.285053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827579.285057:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827579.285061:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 585 00000100:00080000:1.0:1787827579.285064:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827579.285066:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827579.285069:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827579.285080:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827579.285081:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517763200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827579.285094:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff8b2e88077c00 x1874672517762560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.285104:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517762560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.285110:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.285112:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827579.285114:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517762560:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827579.285118:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517762688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827579.285130:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827579.285136:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff8b2e88077480 x1874672517762688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.285148:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517762688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.285153:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.285155:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827579.285157:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517762688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827579.285161:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517762816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827579.285170:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827579.285176:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff8b2e88075680 x1874672517762816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.285183:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517762816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.285188:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.285189:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827579.285191:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517762816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827579.285194:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517762944:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827579.285203:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827579.285209:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff8b2e88074f00 x1874672517762944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.285216:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517762944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.285225:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.285227:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827579.285229:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517762944:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827579.285247:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517763072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827579.285259:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827579.285266:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff8b2e88076940 x1874672517763072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.285273:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517763072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.285278:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.285280:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827579.285282:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517763072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827579.285285:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517763200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827579.285294:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827579.285299:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827579/real 1787827579] req@ffff8b2e88074780 x1874672517763200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827579.285306:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517763200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827579.285311:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827579.285313:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827579.285315:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517763200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827580.308840:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827581.332766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827582.164823:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827582.356770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827583.188840:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827583.380793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827584.212934:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827584.404765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827584.404991:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517763328:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827584.405076:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827584.405120:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff8b2e88041e00 x1874672517763328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827584.405154:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517763328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827584.405188:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827584.405194:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827584.405205:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517763328:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827584.405249:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517763456:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827584.405291:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827584.405311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff8b2e880403c0 x1874672517763456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827584.405329:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672517763456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827584.405339:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827584.405344:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827584.405349:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517763456:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827584.405356:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517763584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827584.405374:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827584.405383:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff8b2e880430c0 x1874672517763584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827584.405390:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517763584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827584.405396:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827584.405398:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827584.405401:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517763584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827584.405406:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517763712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827584.405423:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827584.405434:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff8b2e88040000 x1874672517763712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827584.405447:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517763712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827584.405456:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827584.405459:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827584.405461:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517763712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827584.405466:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517763840:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827584.405481:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827584.405489:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff8b2e88040f00 x1874672517763840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827584.405496:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517763840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827584.405502:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827584.405504:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827584.405506:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517763840:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827584.405518:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517763968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827584.405528:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827584.405534:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff8b2e88042940 x1874672517763968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827584.405542:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517763968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827584.405546:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827584.405549:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827584.405550:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517763968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827584.405554:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517764096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827584.405563:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827584.405568:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827584/real 1787827584] req@ffff8b2e88043480 x1874672517764096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827584.405575:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517764096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827584.405581:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827584.405583:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827584.405585:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517764096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827585.428731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827586.452760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827587.284926:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827587.476868:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827588.308900:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827588.500759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827589.332852:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827589.524778:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827589.524779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827589.524790:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827589.524796:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827589.524807:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 595 00000100:00080000:1.0:1787827589.524819:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827589.524827:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827589.524890:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517764224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827589.524922:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827589.524927:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827589.524930:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827589.524935:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 595 00000100:00080000:1.0:1787827589.524940:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827589.524943:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827589.524951:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517764224:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787827589.524954:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827589.524967:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517764352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827589.524977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827589.524980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827589.524981:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827589.524985:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 595 00000100:00080000:1.0:1787827589.524987:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827589.524990:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827589.524993:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827589.525001:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517764480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827589.525007:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827589.525010:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827589.525011:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827589.525014:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 595 00000100:00080000:1.0:1787827589.525017:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827589.525019:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:2.0:1787827589.525023:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787827589.525024:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827589.525034:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517764608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827589.525042:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827589.525045:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827589.525047:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827589.525049:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 595 00000100:00080000:1.0:1787827589.525051:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827589.525054:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827589.525057:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827589.525065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517764736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827589.525071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827589.525075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787827589.525076:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff8b2e880743c0 x1874672517764224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827589.525078:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827589.525085:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 595 00000100:00080000:1.0:1787827589.525087:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827589.525090:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827589.525092:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827589.525103:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517764864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827589.525111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827589.525111:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517764224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827589.525120:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827589.525123:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827589.525132:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 595 00000100:00080000:2.0:1787827589.525134:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827589.525136:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827589.525141:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787827589.525144:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827589.525149:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517764224:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787827589.525150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827589.525161:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517764992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827589.525178:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517764352:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827589.525197:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827589.525207:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff8b2e88077840 x1874672517764352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827589.525216:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517764352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827589.525227:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827589.525229:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827589.525232:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517764352:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827589.525236:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517764480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827589.525248:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827589.525254:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff8b2e880752c0 x1874672517764480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827589.525262:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517764480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827589.525267:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827589.525269:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827589.525271:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517764480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827589.525275:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517764608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827589.525285:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827589.525291:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff8b2e88074000 x1874672517764608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827589.525299:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517764608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827589.525304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827589.525305:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827589.525308:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517764608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827589.525312:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517764736:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827589.525329:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827589.525336:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff8b2e88075e00 x1874672517764736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827589.525343:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517764736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827589.525348:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827589.525350:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827589.525352:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517764736:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827589.525356:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517764864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827589.525366:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827589.525372:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff8b2e88076d00 x1874672517764864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827589.525379:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517764864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827589.525385:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827589.525386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827589.525388:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517764864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827589.525392:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517764992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827589.525402:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827589.525408:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827589/real 1787827589] req@ffff8b2e880770c0 x1874672517764992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827589.525416:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517764992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827589.525421:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827589.525423:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827589.525425:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517764992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827590.548771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827591.572765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827592.404819:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827592.596759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827593.428866:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827593.620787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827594.452938:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827594.644789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827594.644790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827594.645002:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517765120:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827594.645186:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517765248:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827594.645264:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517765376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827594.645319:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517765504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827594.645370:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517765632:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827594.645409:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517765760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827594.645446:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517765888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827595.668795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827595.668811:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827596.692746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827596.692746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827597.524906:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827597.716800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:1.0:1787827597.717121:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787827597.717188:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827597.717216:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827597.717279:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827597.717295:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827597.717340:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827597.717358:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:2.0:1787827597.717366:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827597] req@ffff8b2e880392c0 x1874672517765248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787827597.717388:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827597.717402:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:2.0:1787827597.717446:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517765248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827597.717483:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827597.717493:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827597.717500:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517765248:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827597.717514:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827597] req@ffff8b2e8803a580 x1874672517765376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827597.717539:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517765376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827597.717554:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827597.717561:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827597.717566:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517765376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827597.717578:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827597] req@ffff8b2e8803a1c0 x1874672517765504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827597.717607:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517765504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827597.717629:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827597.717635:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827597.717640:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517765504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827597.717682:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827597] req@ffff8b2e8803b480 x1874672517765632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827597.717706:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517765632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827597.717720:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827597.717728:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827597.717734:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517765632:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827597.717747:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827597] req@ffff8b2e8803b840 x1874672517765120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827597.717771:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517765120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827597.717784:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827597.717789:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827597.717795:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517765120:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827597.717872:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827597] req@ffff8b2e88038f00 x1874672517765760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827597.717893:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517765760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827597.717906:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827597.717912:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827597.717917:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517765760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827597.717926:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827594/real 1787827597] req@ffff8b2e88039a40 x1874672517765888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827597.717945:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517765888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827597.717956:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827597.717961:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827597.717967:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517765888:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787827598.548925:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827598.740777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827599.572958:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827599.764923:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827599.765080:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517766016:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827599.765204:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827599.765288:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff8b2e88041a40 x1874672517766016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827599.765346:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517766016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827599.765372:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827599.765384:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827599.765394:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517766016:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827599.765435:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517766144:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827599.765491:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827599.765522:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff8b2e88043c00 x1874672517766144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827599.765559:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043c00 x1874672517766144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827599.765606:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827599.765620:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827599.765632:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517766144:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827599.765709:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517766272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827599.765761:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827599.765792:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff8b2e88043480 x1874672517766272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827599.765832:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517766272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827599.765845:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827599.765850:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827599.765854:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517766272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827599.765863:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517766400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827599.765890:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827599.765907:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff8b2e88042940 x1874672517766400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827599.765922:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517766400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827599.765931:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827599.765935:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827599.765940:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517766400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827599.765969:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517766528:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827599.765996:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827599.766017:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff8b2e88040f00 x1874672517766528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827599.766032:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517766528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827599.766041:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827599.766044:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827599.766048:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517766528:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827599.766055:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517766656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827599.766080:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827599.766095:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff8b2e88040000 x1874672517766656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827599.766110:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517766656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827599.766120:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827599.766124:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827599.766127:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517766656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827599.766136:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517766784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827599.766160:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827599.766177:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827599/real 1787827599] req@ffff8b2e880430c0 x1874672517766784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827599.766192:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517766784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827624 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827599.766201:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827599.766205:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827599.766209:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517766784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827600.788933:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827601.812764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827602.644845:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827602.836719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827603.668886:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827603.860756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827604.692845:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827604.884729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827604.884902:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517766912:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827604.884993:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827604.885056:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff8b2e880403c0 x1874672517766912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827604.885080:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672517766912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827604.885104:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827604.885114:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827604.885122:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517766912:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827604.885151:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517767040:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827604.885172:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827604.885182:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff8b2e88041e00 x1874672517767040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827604.885190:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517767040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827604.885201:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827604.885204:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827604.885206:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517767040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827604.885210:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517767168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827604.885224:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827604.885235:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff8b2e88040b40 x1874672517767168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827604.885248:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517767168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827604.885257:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827604.885271:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827604.885275:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517767168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827604.885280:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517767296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827604.885297:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827604.885306:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff8b2e880421c0 x1874672517767296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827604.885314:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517767296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827604.885319:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827604.885322:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827604.885325:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517767296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827604.885329:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517767424:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827604.885340:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827604.885345:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff8b2e88042d00 x1874672517767424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827604.885353:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517767424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827604.885359:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827604.885360:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827604.885362:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517767424:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827604.885365:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517767552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827604.885376:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827604.885382:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff8b2e88043840 x1874672517767552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827604.885389:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517767552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827604.885394:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827604.885396:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827604.885398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517767552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827604.885401:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517767680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827604.885411:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827604.885417:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827604/real 1787827604] req@ffff8b2e88040780 x1874672517767680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827604.885424:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517767680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827629 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827604.885429:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827604.885431:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827604.885433:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517767680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827605.908804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827606.932765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827607.764816:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827607.956796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827608.788806:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827608.980820:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827609.812919:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827610.004804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827610.005060:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517767808:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787827610.005186:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827610.005281:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827610/real 1787827610] req@ffff8b2e880412c0 x1874672517767808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827610.005350:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672517767808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827610.005374:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827610.005391:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827610.005405:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517767808:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827610.005457:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517767936:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827610.005517:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827610.005549:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827610/real 1787827610] req@ffff8b2e88042580 x1874672517767936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827610.005630:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672517767936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827610.005745:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827610.005753:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827610.005759:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517767936:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827610.005768:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517768064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827610.005804:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827610.005817:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827610/real 1787827610] req@ffff8b2e88040780 x1874672517768064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827610.005830:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517768064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827610.005839:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827610.005843:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827610.005846:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517768064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827610.005852:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517768192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827610.005867:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827610.005877:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827610/real 1787827610] req@ffff8b2e88043840 x1874672517768192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827610.005888:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517768192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827610.005896:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827610.005899:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827610.005903:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517768192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827610.005910:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517768320:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827610.005926:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827610.005935:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827610/real 1787827610] req@ffff8b2e88042d00 x1874672517768320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827610.005946:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517768320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827610.005954:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827610.005957:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827610.005960:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517768320:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827610.005979:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517768448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827610.005996:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827610.006006:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827610/real 1787827610] req@ffff8b2e880421c0 x1874672517768448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827610.006018:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517768448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827610.006026:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827610.006029:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827610.006032:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517768448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827610.006038:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517768576:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827610.006054:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827610.006064:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827610/real 1787827610] req@ffff8b2e88040b40 x1874672517768576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827610.006076:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517768576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827610.006084:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827610.006087:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827610.006090:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517768576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827611.028855:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827612.052759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827612.884972:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827613.076758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827613.908815:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827614.100778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827614.932932:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827615.124784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827615.124845:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827615.124853:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827615.124859:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827615.124895:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 621 00000100:00080000:1.0:1787827615.124907:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827615.124913:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827615.124969:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517768704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827615.125005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827615.125009:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827615.125011:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827615.125016:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 621 00000100:00080000:1.0:1787827615.125020:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827615.125024:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827615.125034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827615.125047:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517768832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827615.125060:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827615.125063:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827615.125065:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827615.125069:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 621 00000100:00080000:1.0:1787827615.125072:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827615.125083:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827615.125085:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517768704:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787827615.125089:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827615.125115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517768960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827615.125126:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827615.125128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827615.125130:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827615.125136:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 621 00000100:00080000:1.0:1787827615.125140:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827615.125143:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827615.125147:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827615.125159:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517769088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827615.125168:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827615.125171:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827615.125175:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827615.125179:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 621 00000100:00080000:1.0:1787827615.125182:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827615.125185:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827615.125191:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827615.125199:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517769216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827615.125208:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827615.125211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827615.125245:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827615.125246:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827615.125258:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 621 00000100:00080000:1.0:1787827615.125262:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827615.125268:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827615.125274:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827615.125285:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517769344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827615.125299:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827615.125302:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827615.125305:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827615.125310:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 621 00000100:00080000:1.0:1787827615.125313:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827615.125318:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827615.125322:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827615.125329:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827615/real 1787827615] req@ffff8b2e88076580 x1874672517768704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827615.125336:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517769472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827615.125419:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517768704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827615.125432:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827615.125453:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827615.125457:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517768704:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827615.125530:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517768832:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827615.125589:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827615.125601:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827615/real 1787827615] req@ffff8b2e880761c0 x1874672517768832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827615.125611:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517768832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827615.125634:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827615.125637:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827615.125640:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517768832:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827615.125646:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517768960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827615.125661:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827615.125669:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827615/real 1787827615] req@ffff8b2e88074b40 x1874672517768960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827615.125728:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517768960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827615.125748:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827615.125751:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827615.125754:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517768960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827615.125760:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517769088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827615.125779:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827615.125789:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827615/real 1787827615] req@ffff8b2e880770c0 x1874672517769088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827615.125798:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517769088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827615.125805:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827615.125807:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827615.125811:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517769088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827615.125816:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517769216:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827615.125829:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827615.125836:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827615/real 1787827615] req@ffff8b2e88076d00 x1874672517769216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827615.125845:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517769216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827615.125852:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827615.125854:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827615.125856:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517769216:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827615.125861:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517769344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827615.125874:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827615.125891:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827615/real 1787827615] req@ffff8b2e88075e00 x1874672517769344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827615.125900:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517769344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827615.125906:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827615.125908:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827615.125910:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517769344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827615.125915:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517769472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827615.125927:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827615.125934:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827615/real 1787827615] req@ffff8b2e88074000 x1874672517769472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827615.125953:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517769472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827615.125959:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827615.125961:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827615.125963:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517769472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827616.148827:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827617.172800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827618.004880:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827618.196818:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827619.028907:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827619.220805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827620.052863:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827620.244765:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827620.244776:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827620.244782:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827620.244792:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 626 00000100:00080000:1.0:1787827620.244808:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827620.244815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787827620.244816:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827620.244856:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517769600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827620.244882:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827620.244885:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827620.244887:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827620.244891:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 626 00000100:00080000:1.0:1787827620.244893:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827620.244896:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827620.244905:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827620.244917:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517769728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827620.244925:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827620.244928:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827620.244936:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827620.244938:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517769600:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827620.244953:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 626 00000100:00080000:1.0:1787827620.244958:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827620.244961:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827620.244964:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827620.244977:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517769856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827620.244984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827620.244986:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827620.244987:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827620.244990:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 626 00000100:00080000:1.0:1787827620.244993:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827620.244997:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827620.245000:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827620.245011:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517769984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827620.245018:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827620.245020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827620.245021:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827620.245026:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 626 00000100:00080000:1.0:1787827620.245037:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827620.245040:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827620.245046:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787827620.245056:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787827620.245064:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517770112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827620.245073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827620.245076:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827620.245078:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827620.245082:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 626 00000100:00080000:1.0:1787827620.245104:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827620.245107:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827620.245112:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827620.245123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517770240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827620.245130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827620.245133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827620.245135:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827620.245138:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff8b2e880752c0 x1874672517769600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827620.245140:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 626 00000100:00080000:1.0:1787827620.245147:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827620.245164:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827620.245180:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827620.245192:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517769600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827620.245194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517770368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827620.245282:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827620.245290:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827620.245296:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517769600:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787827620.245326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517769728:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827620.245354:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827620.245368:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff8b2e88077840 x1874672517769728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827620.245381:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517769728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827620.245408:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827620.245412:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827620.245416:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517769728:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827620.245422:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517769856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827620.245443:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827620.245460:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff8b2e880743c0 x1874672517769856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827620.245480:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517769856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827620.245494:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827620.245499:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827620.245505:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517769856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827620.245516:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517769984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827620.245546:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827620.245559:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff8b2e88074780 x1874672517769984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827620.245572:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517769984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827620.245581:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827620.245584:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827620.245589:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517769984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827620.245595:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517770112:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827620.245627:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827620.245638:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff8b2e88076940 x1874672517770112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827620.245738:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517770112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827620.245747:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827620.245750:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827620.245753:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517770112:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827620.245760:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517770240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827620.245783:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827620.245794:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff8b2e88074f00 x1874672517770240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827620.245805:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517770240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827620.245812:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827620.245815:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827620.245818:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517770240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827620.245824:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517770368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827620.245837:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827620.245846:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827620/real 1787827620] req@ffff8b2e88075680 x1874672517770368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827620.245857:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517770368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827620.245864:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827620.245866:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827620.245869:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517770368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827621.268730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827622.292780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827623.124876:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827623.316795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827624.148855:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827624.340806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827625.172923:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827625.364769:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827625.364780:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827625.364787:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827625.364800:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 631 00000100:00080000:1.0:1787827625.364815:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827625.364820:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787827625.364829:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827625.364880:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517770496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827625.364910:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827625.364914:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827625.364916:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827625.364920:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 631 00000100:00080000:1.0:1787827625.364925:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827625.364929:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827625.364941:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827625.364960:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517770624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827625.364962:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517770496:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827625.364990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827625.364993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827625.364995:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827625.364999:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 631 00000100:00080000:1.0:1787827625.365005:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827625.365008:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827625.365012:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827625.365021:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517770752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827625.365028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827625.365031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827625.365032:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827625.365035:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 631 00000100:00080000:1.0:1787827625.365038:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827625.365041:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827625.365045:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827625.365055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517770880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827625.365067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787827625.365069:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827625.365073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827625.365078:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827625.365084:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 631 00000100:00080000:1.0:1787827625.365089:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827625.365093:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827625.365097:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827625.365104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517771008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827625.365113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827625.365117:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827625.365119:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827625.365124:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 631 00000100:00080000:1.0:1787827625.365129:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827625.365134:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827625.365141:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff8b2e88077480 x1874672517770496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827625.365143:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827625.365165:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517771136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827625.365176:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827625.365177:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517770496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827625.365187:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827625.365193:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827625.365209:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 631 00000100:00080000:2.0:1787827625.365213:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827625.365215:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827625.365226:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787827625.365227:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787827625.365243:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827625.365244:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517770496:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827625.365259:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517771264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827625.365279:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517770624:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827625.365304:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827625.365315:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff8b2e88077c00 x1874672517770624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827625.365325:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517770624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827625.365332:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827625.365335:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827625.365337:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517770624:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827625.365342:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517770752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827625.365355:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827625.365363:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff8b2e88075a40 x1874672517770752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827625.365372:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517770752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827625.365378:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827625.365381:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827625.365383:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517770752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827625.365387:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517770880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827625.365398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827625.365405:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff8b2e88075680 x1874672517770880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827625.365413:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517770880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827625.365420:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827625.365422:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827625.365424:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517770880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827625.365429:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517771008:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827625.365440:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827625.365447:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff8b2e88074f00 x1874672517771008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827625.365455:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517771008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827625.365461:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827625.365463:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827625.365465:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517771008:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827625.365470:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517771136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827625.365481:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827625.365488:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff8b2e88076940 x1874672517771136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827625.365496:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517771136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827625.365502:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827625.365504:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827625.365507:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517771136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827625.365511:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517771264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827625.365522:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827625.365529:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827625/real 1787827625] req@ffff8b2e88074780 x1874672517771264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827625.365537:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517771264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827625.365543:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827625.365545:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827625.365547:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517771264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827626.388756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827627.412817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827628.244818:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827629.268832:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827630.292831:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827630.484795:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827630.484807:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827630.484814:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827630.484830:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 636 00000100:00080000:1.0:1787827630.484850:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827630.484862:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827630.484925:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517771392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827630.484985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827630.484990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827630.484993:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827630.484998:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 636 00000100:00080000:1.0:1787827630.485003:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827630.485010:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827630.485042:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827630.485055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517771520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827630.485067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827630.485074:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517771392:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827630.485075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827630.485078:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827630.485110:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 636 00000100:00080000:1.0:1787827630.485115:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827630.485121:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827630.485128:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827630.485145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517771648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827630.485158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827630.485163:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827630.485166:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827630.485170:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 636 00000100:00080000:1.0:1787827630.485183:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827630.485187:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:2.0:1787827630.485193:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787827630.485195:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827630.485210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517771776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827630.485218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827630.485222:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827630.485224:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827630.485228:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 636 00000100:00080000:1.0:1787827630.485231:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827630.485235:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827630.485238:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827630.485249:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517771904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827630.485256:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff8b2e880743c0 x1874672517771392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827630.485266:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827630.485273:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827630.485277:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827630.485285:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 636 00000100:00080000:1.0:1787827630.485288:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827630.485292:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827630.485296:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827630.485301:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880743c0 x1874672517771392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827630.485307:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517772032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827630.485334:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827630.485337:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827630.485338:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827630.485343:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827630.485346:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787827630.485352:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 636 00000100:00080000:1.0:1787827630.485355:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787827630.485357:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880743c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517771392:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787827630.485362:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827630.485377:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827630.485385:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517772160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827630.485394:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517771520:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827630.485425:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827630.485439:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff8b2e88077840 x1874672517771520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827630.485450:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517771520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827630.485457:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827630.485460:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827630.485463:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517771520:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827630.485467:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517771648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827630.485482:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827630.485490:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff8b2e880752c0 x1874672517771648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827630.485499:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517771648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827630.485506:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827630.485508:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827630.485510:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517771648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827630.485515:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517771776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827630.485527:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827630.485534:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff8b2e88074000 x1874672517771776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827630.485550:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517771776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827630.485556:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827630.485558:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827630.485560:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517771776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827630.485564:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517771904:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827630.485576:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827630.485583:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff8b2e88075e00 x1874672517771904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827630.485616:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517771904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827630.485623:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827630.485626:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827630.485628:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517771904:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827630.485633:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517772032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827630.485682:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827630.485699:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff8b2e88076d00 x1874672517772032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827630.485715:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517772032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827630.485734:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827630.485739:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827630.485743:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517772032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827630.485769:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517772160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827630.485794:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827630.485805:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827630/real 1787827630] req@ffff8b2e880770c0 x1874672517772160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827630.485814:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517772160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827630.485822:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827630.485824:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827630.485827:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517772160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787827631.508820:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827632.532726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827633.364880:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827633.556756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827634.388861:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827634.580810:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827635.412812:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827635.604789:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827635.604799:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787827635.604804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827635.604806:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827635.604822:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 642 00000100:00080000:1.0:1787827635.604836:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827635.604845:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827635.604894:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517772288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827635.604921:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827635.604924:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827635.604926:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827635.604930:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 642 00000100:00080000:1.0:1787827635.604933:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827635.604936:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827635.604948:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827635.604960:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517772288:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827635.604967:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517772416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827635.604984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827635.604989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827635.604992:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827635.604997:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 642 00000100:00080000:1.0:1787827635.605003:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827635.605009:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827635.605017:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827635.605029:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517772544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827635.605038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827635.605041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827635.605045:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787827635.605046:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787827635.605051:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 642 00000100:00080000:1.0:1787827635.605056:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827635.605060:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827635.605065:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827635.605075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517772672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827635.605083:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827635.605086:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827635.605087:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827635.605092:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 642 00000100:00080000:1.0:1787827635.605095:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827635.605098:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827635.605102:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827635.605104:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827635] req@ffff8b2e88074b40 x1874672517772288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827635.605113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517772800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827635.605124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827635.605127:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827635.605129:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827635.605134:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 642 00000100:00080000:1.0:1787827635.605137:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827635.605142:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827635.605145:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517772288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787827635.605146:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827635.605157:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517772928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827635.605171:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827635.605173:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787827635.605174:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827635.605178:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827635.605185:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 642 00000100:00080000:2.0:1787827635.605186:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787827635.605188:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827635.605195:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827635.605199:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827635.605203:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517772288:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787827635.605210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517773056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827635.605236:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517772416:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827635.605262:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827635.605275:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827635] req@ffff8b2e880761c0 x1874672517772416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827635.605286:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517772416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827635.605293:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827635.605296:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827635.605299:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517772416:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827635.605303:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517772544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827635.605316:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827635.605324:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827635] req@ffff8b2e88076580 x1874672517772544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827635.605333:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517772544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827635.605339:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827635.605341:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827635.605343:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517772544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827635.605348:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517772672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827635.605360:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827635.605368:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827635] req@ffff8b2e880770c0 x1874672517772672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827635.605377:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517772672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827635.605383:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827635.605385:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827635.605387:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517772672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827635.605391:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517772800:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787827635.605403:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827635.605410:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827635] req@ffff8b2e88076d00 x1874672517772800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827635.605419:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517772800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827635.605425:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827635.605427:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827635.605429:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517772800:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787827635.605434:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517772928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827635.605445:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827635.605452:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827635] req@ffff8b2e88075e00 x1874672517772928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827635.605461:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517772928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827635.605467:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827635.605470:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827635.605472:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517772928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827635.605476:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517773056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787827635.605488:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787827635.605494:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787827635/real 1787827635] req@ffff8b2e88074000 x1874672517773056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827635.605502:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517773056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787827660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787827635.605508:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787827635.605510:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787827635.605512:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517773056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787827636.628757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827637.652763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827638.484898:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827638.676854:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827639.508861:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827639.700806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827640.532816:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827640.724810:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787827640.724831:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827640.724843:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827640.724848:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827640.724859:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 647 00000100:00080000:2.0:1787827640.724872:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827640.724880:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827640.724926:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517773184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827640.724976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827640.724979:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827640.724981:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827640.724985:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 647 00000100:00080000:2.0:1787827640.724987:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827640.724990:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827640.724999:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827640.725009:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517773312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827640.725018:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827640.725021:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827640.725023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827640.725028:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 647 00000100:00080000:2.0:1787827640.725031:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827640.725036:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827640.725042:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827640.725048:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517773440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827640.725054:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827640.725057:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827640.725058:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827640.725060:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 647 00000100:00080000:2.0:1787827640.725063:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827640.725066:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827640.725070:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827640.725079:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517773568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827640.725085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827640.725087:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827640.725089:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827640.725092:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 647 00000100:00080000:2.0:1787827640.725102:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827640.725105:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827640.725107:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827640.725117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517773696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827640.725125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827640.725128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827640.725130:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827640.725133:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 647 00000100:00080000:2.0:1787827640.725136:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827640.725138:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827640.725141:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827640.725147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517773824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827640.725158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827640.725161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827640.725162:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827640.725165:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 647 00000100:00080000:2.0:1787827640.725167:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827640.725169:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827640.725172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827640.725181:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517773952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827641.748747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827642.772770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827643.604848:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827643.796748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827644.820736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827645.652867:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827645.844747:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827645.844748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827645.844761:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827645.844767:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.844779:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 652 00000100:00080000:1.0:1787827645.844793:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827645.844803:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827645.844850:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517774080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827645.844880:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.844884:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827645.844886:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.844893:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 652 00000100:00080000:1.0:1787827645.844898:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827645.844903:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827645.844917:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827645.844934:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517774208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827645.844947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.844952:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827645.844954:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.844958:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 652 00000100:00080000:1.0:1787827645.844963:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827645.844967:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827645.844974:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827645.844987:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517774336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827645.845003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.845008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827645.845010:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.845016:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 652 00000100:00080000:1.0:1787827645.845022:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827645.845027:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827645.845032:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827645.845044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517774464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827645.845058:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.845065:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827645.845069:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.845080:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 652 00000100:00080000:1.0:1787827645.845084:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827645.845089:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827645.845097:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827645.845130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517774592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827645.845143:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.845147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827645.845150:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.845156:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 652 00000100:00080000:1.0:1787827645.845162:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827645.845171:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827645.845178:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827645.845193:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517774720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827645.845213:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827645.845218:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827645.845220:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827645.845228:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 652 00000100:00080000:1.0:1787827645.845235:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827645.845243:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827645.845251:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827645.845267:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517774848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827646.868730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827647.892745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827648.724884:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827650.772885:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827650.964773:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827650.964783:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827650.964789:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827650.964802:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 657 00000100:00080000:1.0:1787827650.964816:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827650.964827:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827650.964878:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517774976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827650.964909:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827650.964913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827650.964916:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827650.964922:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 657 00000100:00080000:1.0:1787827650.964927:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827650.964933:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827650.964948:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827650.964967:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517775104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827650.964981:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827650.964985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827650.964988:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827650.964992:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 657 00000100:00080000:1.0:1787827650.965005:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827650.965013:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827650.965021:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827650.965035:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517775232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827650.965049:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827650.965053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827650.965055:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827650.965061:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 657 00000100:00080000:1.0:1787827650.965067:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827650.965076:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827650.965088:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827650.965108:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517775360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827650.965131:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827650.965136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827650.965138:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827650.965144:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 657 00000100:00080000:1.0:1787827650.965150:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827650.965160:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827650.965166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827650.965185:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517775488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827650.965215:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827650.965224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827650.965230:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827650.965238:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 657 00000100:00080000:1.0:1787827650.965246:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827650.965254:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827650.965264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827650.965281:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517775616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827650.965294:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827650.965301:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827650.965306:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827650.965312:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 657 00000100:00080000:1.0:1787827650.965319:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827650.965328:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827650.965339:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827650.965361:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517775744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827653.844827:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827655.892897:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827656.084738:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827656.084752:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827656.084758:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827656.084766:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 662 00000100:00080000:1.0:1787827656.084776:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827656.084783:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827656.084817:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517775872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827656.084844:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827656.084848:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827656.084850:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827656.084855:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 662 00000100:00080000:1.0:1787827656.084860:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827656.084865:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827656.084875:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827656.084890:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517776000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827656.084899:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827656.084902:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827656.084903:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827656.084905:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 662 00000100:00080000:1.0:1787827656.084908:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827656.084911:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827656.084914:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827656.084923:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517776128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827656.084929:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827656.084931:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827656.084932:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827656.084935:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 662 00000100:00080000:1.0:1787827656.084937:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827656.084940:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827656.084943:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827656.084953:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517776256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827656.084964:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827656.084967:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827656.084968:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827656.084971:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 662 00000100:00080000:1.0:1787827656.084973:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827656.084975:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827656.084978:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827656.084986:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517776384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827656.084992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827656.084995:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827656.084997:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827656.085004:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 662 00000100:00080000:1.0:1787827656.085009:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827656.085014:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827656.085020:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827656.085031:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517776512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827656.085038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827656.085040:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827656.085042:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827656.085047:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 662 00000100:00080000:1.0:1787827656.085053:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827656.085059:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827656.085066:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827656.085074:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517776640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827658.964823:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827661.012842:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827661.204774:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827661.204783:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827661.204789:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827661.204803:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 667 00000100:00080000:1.0:1787827661.204819:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827661.204829:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827661.204880:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517776768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827661.204911:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827661.204915:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827661.204917:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827661.204922:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 667 00000100:00080000:1.0:1787827661.204928:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827661.204934:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827661.204948:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827661.204968:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517776896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827661.204988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827661.204993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827661.204996:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827661.205000:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 667 00000100:00080000:1.0:1787827661.205007:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827661.205011:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827661.205018:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827661.205032:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517777024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827661.205043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827661.205047:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827661.205050:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827661.205054:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 667 00000100:00080000:1.0:1787827661.205059:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827661.205068:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827661.205077:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827661.205093:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517777152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827661.205105:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827661.205112:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827661.205117:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827661.205124:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 667 00000100:00080000:1.0:1787827661.205130:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827661.205135:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827661.205142:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827661.205158:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517777280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827661.205171:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827661.205175:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827661.205177:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827661.205186:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 667 00000100:00080000:1.0:1787827661.205193:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827661.205199:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827661.205204:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827661.205221:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517777408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827661.205241:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827661.205246:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827661.205249:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827661.205255:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 667 00000100:00080000:1.0:1787827661.205264:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827661.205269:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827661.205276:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827661.205291:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517777536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827664.084866:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000400:00000100:2.0:1787827664.277028:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787827664.277083:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827664.277104:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827664.277151:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827664.277160:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827664.277170:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827664.277181:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827664.277190:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827664.277199:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00010000:00010000:1.0:1787827666.132874:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827666.324895:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827667.348860:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827668.372795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827669.204845:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827671.252849:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827671.444811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827672.468787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827673.492755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827674.324835:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827674.516740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827675.540758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827676.372774:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827676.564808:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827676.564817:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827676.564824:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827676.564858:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 682 00000100:00080000:2.0:1787827676.564874:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827676.564884:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827676.564936:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d2c0 x1874672517779456/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827676.564969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827676.564973:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827676.564976:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827676.564980:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 682 00000100:00080000:2.0:1787827676.564984:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827676.564989:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827676.565002:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827676.565016:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517779584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827676.565029:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827676.565038:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827676.565044:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827676.565054:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 682 00000100:00080000:2.0:1787827676.565058:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827676.565062:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827676.565067:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827676.565083:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517779712/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827676.565093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827676.565096:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827676.565114:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827676.565119:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 682 00000100:00080000:2.0:1787827676.565124:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827676.565128:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827676.565133:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827676.565144:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517779840/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827676.565152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827676.565156:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827676.565161:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827676.565189:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 682 00000100:00080000:2.0:1787827676.565194:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827676.565199:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827676.565206:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827676.565220:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517779968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827676.565239:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827676.565243:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827676.565245:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827676.565250:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 682 00000100:00080000:2.0:1787827676.565254:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827676.565258:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827676.565303:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827676.565320:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517780096/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827676.565354:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827676.565358:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827676.565360:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827676.565368:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 682 00000100:00080000:2.0:1787827676.565372:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827676.565384:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827676.565394:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827676.565407:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517780224/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827679.444986:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827681.492859:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827681.684757:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827681.684770:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827681.684775:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827681.684786:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 688 00000100:00080000:2.0:1787827681.684798:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827681.684806:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827681.684860:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517780352/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827681.684909:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827681.684912:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827681.684914:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827681.684917:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 688 00000100:00080000:2.0:1787827681.684920:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827681.684923:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827681.684932:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827681.684943:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517780480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827681.684961:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827681.684964:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827681.684966:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827681.684973:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 688 00000100:00080000:2.0:1787827681.684979:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827681.684981:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827681.684984:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827681.684992:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517780608/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827681.684998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827681.685002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827681.685003:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827681.685006:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 688 00000100:00080000:2.0:1787827681.685008:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827681.685011:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827681.685014:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827681.685032:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517780736/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827681.685037:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827681.685039:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827681.685040:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827681.685043:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 688 00000100:00080000:2.0:1787827681.685046:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827681.685049:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827681.685052:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827681.685062:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517780864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827681.685069:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827681.685072:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827681.685074:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827681.685077:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 688 00000100:00080000:2.0:1787827681.685080:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827681.685093:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827681.685097:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827681.685104:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517780992/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827681.685110:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827681.685112:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827681.685113:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827681.685117:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 688 00000100:00080000:2.0:1787827681.685121:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827681.685129:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827681.685136:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827681.685146:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517781120/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827684.564800:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827686.612890:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827686.804832:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827687.828874:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827688.852751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827689.684881:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827691.732804:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827691.924751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827692.948749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827693.972767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827694.804789:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827694.996746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827696.020715:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827696.852821:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827697.044807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827698.068738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827699.092719:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827699.924781:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787827701.972827:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827702.164774:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827702.164782:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827702.164787:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827702.164799:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 708 00000100:00080000:2.0:1787827702.164811:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827702.164818:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827702.164853:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517783936/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827702.164887:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827702.164890:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827702.164892:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827702.164895:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 708 00000100:00080000:2.0:1787827702.164898:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827702.164900:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827702.164908:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827702.164919:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517784064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827702.164927:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827702.164933:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827702.164935:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827702.164943:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 708 00000100:00080000:2.0:1787827702.164946:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827702.164948:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827702.164952:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827702.164962:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d2c0 x1874672517784192/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827702.164974:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827702.164977:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827702.164978:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827702.164980:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 708 00000100:00080000:2.0:1787827702.164983:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827702.164985:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827702.164989:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827702.165000:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517784320/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827702.165010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827702.165013:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827702.165015:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827702.165019:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 708 00000100:00080000:2.0:1787827702.165022:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827702.165024:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827702.165027:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827702.165036:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517784448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827702.165044:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827702.165053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827702.165056:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827702.165065:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 708 00000100:00080000:2.0:1787827702.165070:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827702.165074:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827702.165077:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827702.165102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517784576/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827702.165117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827702.165123:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827702.165128:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827702.165134:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 708 00000100:00080000:2.0:1787827702.165137:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827702.165139:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827702.165143:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827702.165152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517784704/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827704.212782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827705.044901:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827705.236809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827706.260777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827707.092994:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827707.284772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827707.284779:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827707.284785:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827707.284788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787827707.284797:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 713 00000100:00080000:2.0:1787827707.284814:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827707.284823:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827707.284880:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517784832/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827707.284908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827707.284913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827707.284914:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827707.284918:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 713 00000100:00080000:2.0:1787827707.284932:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827707.284936:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827707.284946:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827707.284960:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517784960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827707.284978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827707.284981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827707.284982:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827707.284996:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 713 00000100:00080000:2.0:1787827707.284998:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827707.285001:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827707.285004:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827707.285015:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517785088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827707.285021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827707.285024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827707.285025:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827707.285028:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 713 00000100:00080000:2.0:1787827707.285031:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827707.285034:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827707.285039:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827707.285046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517785216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827707.285088:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827707.285095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827707.285097:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827707.285102:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 713 00000100:00080000:2.0:1787827707.285106:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827707.285111:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827707.285116:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827707.285126:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517785344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827707.285133:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827707.285136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827707.285137:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827707.285141:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 713 00000100:00080000:2.0:1787827707.285152:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827707.285155:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827707.285159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827707.285170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517785472/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827707.285197:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827707.285206:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827707.285220:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827707.285224:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 713 00000100:00080000:2.0:1787827707.285227:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827707.285230:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827707.285233:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827707.285243:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517785600/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827708.308827:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827709.332715:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827710.164834:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827710.356740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827711.380716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827712.212811:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827712.404740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787827712.404816:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827712.404824:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827712.404829:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827712.404839:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 718 00000100:00080000:2.0:1787827712.404851:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827712.404859:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827712.404899:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d2c0 x1874672517785728/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827712.404923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827712.404926:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827712.404928:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827712.404932:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 718 00000100:00080000:2.0:1787827712.404935:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827712.404955:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827712.404966:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827712.404982:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517785856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827712.405006:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827712.405010:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827712.405012:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827712.405035:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 718 00000100:00080000:2.0:1787827712.405039:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827712.405042:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827712.405048:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827712.405076:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517785984/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827712.405084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827712.405087:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827712.405090:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827712.405093:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 718 00000100:00080000:2.0:1787827712.405096:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827712.405099:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827712.405103:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827712.405132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517786112/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827712.405141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827712.405144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827712.405146:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827712.405149:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 718 00000100:00080000:2.0:1787827712.405152:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827712.405155:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827712.405159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827712.405169:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517786240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827712.405176:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827712.405178:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827712.405182:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827712.405187:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 718 00000100:00080000:2.0:1787827712.405192:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827712.405200:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827712.405207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827712.405215:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517786368/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827712.405226:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827712.405234:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827712.405238:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827712.405246:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 718 00000100:00080000:2.0:1787827712.405249:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827712.405252:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827712.405277:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827712.405288:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517786496/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827713.428753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827714.452831:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827715.284865:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827715.476714:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827716.500719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827717.332784:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827717.524791:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827717.524792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827717.524806:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827717.524829:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827717.524845:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 723 00000100:00080000:1.0:1787827717.524862:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827717.524875:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827717.524936:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517786624/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827717.525002:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827717.525008:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827717.525013:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827717.525021:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 723 00000100:00080000:1.0:1787827717.525025:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827717.525030:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827717.525041:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827717.525057:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517786752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827717.525070:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827717.525078:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827717.525082:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827717.525088:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 723 00000100:00080000:1.0:1787827717.525094:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827717.525099:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827717.525105:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827717.525121:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880743c0 x1874672517786880/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827717.525137:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827717.525148:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827717.525153:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827717.525161:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 723 00000100:00080000:1.0:1787827717.525170:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827717.525178:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827717.525184:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827717.525200:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517787008/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827717.525215:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827717.525220:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827717.525225:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827717.525236:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 723 00000100:00080000:1.0:1787827717.525244:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827717.525251:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827717.525262:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827717.525283:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517787136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827717.525298:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827717.525303:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827717.525307:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827717.525318:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 723 00000100:00080000:1.0:1787827717.525325:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827717.525335:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827717.525345:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827717.525363:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517787264/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827717.525380:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827717.525385:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827717.525387:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827717.525395:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 723 00000100:00080000:1.0:1787827717.525413:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827717.525420:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827717.525425:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827717.525438:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517787392/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827718.548779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827719.572734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827720.404871:0:14480:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827720.596796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827722.452875:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827722.644751:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827722.644757:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827722.644762:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827722.644770:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 729 00000100:00080000:1.0:1787827722.644780:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827722.644785:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827722.644816:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517787520/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827722.644838:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827722.644841:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827722.644842:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827722.644846:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 729 00000100:00080000:1.0:1787827722.644848:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827722.644851:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827722.644860:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827722.644871:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517787648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827722.644880:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827722.644882:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827722.644883:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827722.644886:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 729 00000100:00080000:1.0:1787827722.644888:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827722.644890:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827722.644893:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827722.644901:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517787776/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827722.644913:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827722.644915:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827722.644917:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827722.644919:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 729 00000100:00080000:1.0:1787827722.644921:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827722.644927:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827722.644931:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827722.644940:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517787904/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827722.644961:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827722.644964:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827722.644966:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827722.644970:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 729 00000100:00080000:1.0:1787827722.644975:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827722.645009:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827722.645017:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827722.645031:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517788032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827722.645079:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827722.645083:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827722.645087:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827722.645094:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 729 00000100:00080000:1.0:1787827722.645099:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827722.645106:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827722.645109:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827722.645117:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517788160/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827722.645126:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827722.645130:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827722.645131:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827722.645134:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 729 00000100:00080000:1.0:1787827722.645137:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827722.645140:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827722.645143:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827722.645154:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517788288/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827725.524806:0:14480:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 16 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:1.0:1787827725.524816:0:14480:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e88b2a800/0x61d2feb329d092eb lrc: 2/0,0 mode: CR/CR res: [0x200000401:0x1b:0x0].0x0 bits 0x8/0x0 rrc: 2 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d228d76c expref: -99 pid: 20226 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827725.524845:0:14480:0:(vvp_object.c:117:vvp_conf_set()) [0x200000401:0x1b:0x0]: losing layout lock 00010000:00010000:1.0:1787827725.524865:0:14480:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e90ea0e00/0x61d2feb329d092c1 lrc: 2/0,0 mode: PR/PR res: [0x200000402:0x9:0x0].0x0 bits 0x1b/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d228d711 expref: -99 pid: 19662 timeout: 0 lvb_type: 3 lru_score: 2 lru_type: 0 00000080:00200000:1.0:1787827725.524874:0:14480:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x9:0x0]: losing layout lock 00010000:00010000:1.0:1787827725.524893:0:14480:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2e90ea1c00/0x61d2feb329d091be lrc: 2/0,0 mode: CR/CR res: [0x200000402:0x9:0x0].0x0 bits 0x8/0x0 rrc: 3 type: IBT gid 0 flags: 0x28400000000 nid: local remote: 0xb0788cb3d228d424 expref: -99 pid: 19608 timeout: 0 lvb_type: 3 lru_score: 7 lru_type: 0 00000080:00200000:1.0:1787827725.524904:0:14480:0:(vvp_object.c:117:vvp_conf_set()) [0x200000402:0x9:0x0]: losing layout lock 00010000:00010000:1.0:1787827725.524908:0:14480:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e901ad800 lock: ffff8b2ea0386200/0x61d2feb329d09178 lrc: 2/0,0 mode: CW/CW res: [0x200000401:0x13:0x0].0x0 bits 0x5/0x0 rrc: 3 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xb0788cb3d228d36e expref: -99 pid: 18902 timeout: 0 lvb_type: 0 lru_score: 2 lru_type: 0 00000002:00100000:1.0:1787827725.524992:0:14480:0:(mdc_request.c:964:mdc_close()) @@@ matched open req@ffff8b2e88b2fc00 x1874672516931456/t4294967388(4294967388) o101->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 600/688 e 0 to 0 dl 1787827176 ref 1 fl Complete:RPQU/604/ffffffff rc 0/-1 job:'bash.0' uid:0 gid:0 projid:0 00000020:00100000:1.0:1787827725.525001:0:14480:0:(genops.c:2423:obd_get_mod_rpc_slot()) lustre-MDT0000-mdc-ffff8b2e901ad800: modify RPC slot 1 is allocated opc 35, max 7 00000100:00100000:1.0:1787827725.525011:0:14480:0:(client.c:820:ptlrpc_reassign_next_xid()) @@@ reassign xid req@ffff8b2e880743c0 x1874672517788544/t0(0) o35->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 1 fl New:QU/400/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:0 00000100:00080000:1.0:1787827725.525029:0:14480:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8b2e880743c0 x1874672517788544/t0(0) o35->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:23/10 lens 392/624 e 0 to 0 dl 0 ref 2 fl Rpc:WQU/600/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:0 00000100:00100000:1.0:1787827725.525036:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827726.548793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827726.548815:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827727.572792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827727.764865:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827728.596773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827728.788814:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827729.620763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827729.812746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827730.644733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827730.644763:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827731.668744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:2.0:1787827731.669439:0:2349:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8b2e88038b40 x1874672517789568/t0(0) o103->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:17/18 lens 440/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827732.692775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:2.0:1787827732.884857:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827732.884868:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827732.884891:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827732.884927:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 739 00000100:00080000:2.0:1787827732.884937:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827732.884943:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827732.885001:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517789696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827732.885025:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827732.885028:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827732.885029:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827732.885032:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 739 00000100:00080000:2.0:1787827732.885035:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827732.885037:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827732.885056:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827732.885072:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517789824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827732.885085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827732.885089:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827732.885091:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827732.885095:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 739 00000100:00080000:2.0:1787827732.885098:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827732.885103:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827732.885120:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827732.885153:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517789952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827732.885165:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827732.885169:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827732.885170:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827732.885175:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 739 00000100:00080000:2.0:1787827732.885179:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827732.885184:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827732.885191:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827732.885207:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517790080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827732.885218:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827732.885223:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827732.885225:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827732.885231:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 739 00000100:00080000:2.0:1787827732.885242:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827732.885253:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827732.885262:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827732.885280:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517790208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827732.885292:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827732.885307:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827732.885316:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827732.885322:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 739 00000100:00080000:2.0:1787827732.885331:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827732.885345:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827732.885372:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827732.885394:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517790336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827732.885486:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827732.885492:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827732.885498:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827732.885512:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 739 00000100:00080000:2.0:1787827732.885523:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827732.885533:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827732.885539:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827732.885553:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517790464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827733.716746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827733.716784:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827734.740739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827735.764763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827736.788767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827737.812770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827737.812830:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827738.004810:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827738.004823:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827738.004831:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827738.004858:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 744 00000100:00080000:2.0:1787827738.004872:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827738.004891:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827738.004930:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517790592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827738.004986:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827738.004991:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827738.004993:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827738.004999:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 744 00000100:00080000:2.0:1787827738.005003:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827738.005009:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827738.005037:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827738.005053:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517790720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827738.005071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827738.005074:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827738.005076:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827738.005099:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 744 00000100:00080000:2.0:1787827738.005105:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827738.005110:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827738.005116:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827738.005132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d2c0 x1874672517790848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827738.005145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827738.005150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827738.005166:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827738.005178:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 744 00000100:00080000:2.0:1787827738.005184:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827738.005189:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827738.005195:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827738.005209:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517790976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827738.005220:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827738.005224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827738.005226:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827738.005248:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 744 00000100:00080000:2.0:1787827738.005253:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827738.005260:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827738.005269:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827738.005284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517791104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827738.005366:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827738.005373:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827738.005377:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827738.005392:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 744 00000100:00080000:2.0:1787827738.005396:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827738.005406:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827738.005411:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827738.005423:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517791232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827738.005435:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827738.005448:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827738.005450:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827738.005454:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 744 00000100:00080000:2.0:1787827738.005458:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827738.005461:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827738.005466:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827738.005476:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517791360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827738.836784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827739.028863:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827739.860748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827740.052799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827740.884753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827741.076775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827741.908756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827741.908875:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827742.100730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827742.932740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:2.0:1787827743.124726:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827743.124729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827743.124737:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827743.124740:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827743.124748:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 749 00000100:00080000:2.0:1787827743.124759:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827743.124768:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827743.124814:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517791488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827743.124836:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827743.124839:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827743.124840:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827743.124843:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 749 00000100:00080000:2.0:1787827743.124847:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827743.124852:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827743.124862:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827743.124880:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517791616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827743.124894:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827743.124898:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827743.124900:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827743.124904:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 749 00000100:00080000:2.0:1787827743.124908:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827743.124911:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827743.124915:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827743.124926:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517791744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827743.124942:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827743.124945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827743.124947:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827743.124950:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 749 00000100:00080000:2.0:1787827743.124954:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827743.124956:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827743.124959:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827743.124967:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517791872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827743.124974:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827743.124978:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827743.124982:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827743.124988:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 749 00000100:00080000:2.0:1787827743.124993:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827743.124998:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827743.125002:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827743.125012:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517792000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827743.125028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827743.125033:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827743.125036:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827743.125044:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 749 00000100:00080000:2.0:1787827743.125049:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827743.125054:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827743.125058:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827743.125064:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517792128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827743.125071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827743.125076:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827743.125079:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827743.125083:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 749 00000100:00080000:2.0:1787827743.125086:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827743.125088:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827743.125092:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827743.125104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517792256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827743.956721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827744.148744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827744.980748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827745.172724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827746.004752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827746.004828:0:14479:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 2 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787827746.004837:0:14479:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000b0ee4dff/0x61d2feb329d09300 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x800028400020000 nid: local remote: 0xb0788cb3d228d7a4 expref: -99 pid: 20230 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827746.004860:0:14479:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1b:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787827746.004925:0:14479:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1b:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787827746.004942:0:14479:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 0000000064cb2201/0x61d2feb329d08eb5 lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0xb0788cb3d228cba3 expref: -99 pid: 14484 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827746.004950:0:14479:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x6:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787827746.004953:0:14479:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x6:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787827746.004984:0:14479:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000b0ee4dff/0x61d2feb329d09300 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0xb0788cb3d228d7a4 expref: -99 pid: 20230 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827746.004991:0:14479:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 0000000064cb2201/0x61d2feb329d08eb5 lrc: 1/0,0 mode: --/PR res: [0x280000400:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0xb0788cb3d228cba3 expref: -99 pid: 14484 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827746.004995:0:14479:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787827746.005022:0:14479:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 00000000b0ee4dff/0x61d2feb329d09300 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x7:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->4095) gid 0 flags: 0x804869400020020 nid: local remote: 0xb0788cb3d228d7a4 expref: -99 pid: 20230 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827746.005038:0:14479:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8b2e84abe800 lock: 0000000064cb2201/0x61d2feb329d08eb5 lrc: 0/0,0 mode: --/PR res: [0x280000400:0x4:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0xb0788cb3d228cba3 expref: -99 pid: 14484 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827746.005049:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827746.005051:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827746.005053:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827746.196727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827747.028735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827747.220731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827748.052747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:2.0:1787827748.244746:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827748.244762:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827748.244784:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827748.244785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827748.244796:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 754 00000100:00080000:2.0:1787827748.244821:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827748.244831:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827748.244873:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e1c0 x1874672517792512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827748.244915:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827748.244919:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827748.244921:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827748.244926:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 754 00000100:00080000:2.0:1787827748.244930:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827748.244935:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827748.244947:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827748.244973:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f840 x1874672517792640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827748.244997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827748.245001:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827748.245003:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827748.245008:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 754 00000100:00080000:2.0:1787827748.245012:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827748.245015:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827748.245021:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827748.245044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517792768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827748.245055:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827748.245059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827748.245061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827748.245072:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 754 00000100:00080000:2.0:1787827748.245080:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827748.245085:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827748.245091:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827748.245112:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517792896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827748.245173:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827748.245184:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827748.245193:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827748.245209:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 754 00000100:00080000:2.0:1787827748.245221:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827748.245236:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827748.245248:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827748.245265:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517793024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827748.245286:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827748.245291:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827748.245294:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827748.245300:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 754 00000100:00080000:2.0:1787827748.245306:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827748.245311:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827748.245317:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827748.245337:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517793152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827748.245353:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827748.245367:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827748.245373:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827748.245384:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 754 00000100:00080000:2.0:1787827748.245393:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827748.245404:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827748.245416:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827748.245434:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517793280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827749.076730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827749.268851:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827750.100789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827750.100853:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827750.292738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827751.124862:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827751.316723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827752.148753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827752.340731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827753.172774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:1.0:1787827753.364756:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.364766:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827753.364770:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.364779:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 759 00000100:00080000:1.0:1787827753.364789:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.364796:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827753.364835:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517793408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.364865:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.364869:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827753.364871:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.364876:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 759 00000100:00080000:1.0:1787827753.364880:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.364885:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827753.364896:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827753.364914:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517793536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.364926:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.364928:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827753.364930:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.364933:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 759 00000100:00080000:1.0:1787827753.364950:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.364953:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827753.364958:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827753.364971:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517793664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.364980:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.364982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827753.364984:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.364987:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 759 00000100:00080000:1.0:1787827753.364989:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.364992:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827753.365004:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827753.365024:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517793792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.365044:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.365051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827753.365054:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.365059:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 759 00000100:00080000:1.0:1787827753.365063:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.365067:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827753.365073:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827753.365089:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517793920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.365097:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.365102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827753.365104:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.365108:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 759 00000100:00080000:1.0:1787827753.365111:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.365117:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827753.365126:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827753.365146:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517794048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827753.365171:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827753.365175:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827753.365177:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827753.365184:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 759 00000100:00080000:1.0:1787827753.365188:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827753.365195:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827753.365204:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827753.365221:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517794176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827754.196788:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827754.196891:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827754.388805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827755.220775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827755.412801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827756.244744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827756.436733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827757.268806:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827757.460762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827758.292808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827758.292850:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827758.484723:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827759.316760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827759.316827:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827759.508817:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827760.340741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827760.532744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827761.364769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827761.364830:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827761.556828:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827762.388742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827762.388795:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827762.580800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827763.412754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827763.604766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787827763.604784:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827763.604792:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827763.604797:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827763.604809:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 770 00000100:00080000:2.0:1787827763.604823:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827763.604830:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827763.604874:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517795200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827763.604903:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827763.604906:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827763.604908:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827763.604912:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 770 00000100:00080000:2.0:1787827763.604916:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827763.604919:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827763.604930:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827763.604939:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517795328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827763.604951:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827763.604959:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827763.604961:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827763.604964:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 770 00000100:00080000:2.0:1787827763.604966:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827763.604969:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827763.604973:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827763.604986:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517795456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827763.604993:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827763.604997:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827763.604998:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827763.605002:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 770 00000100:00080000:2.0:1787827763.605012:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827763.605016:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827763.605020:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827763.605030:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517795584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827763.605039:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827763.605042:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827763.605044:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827763.605048:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 770 00000100:00080000:2.0:1787827763.605051:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827763.605055:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827763.605058:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827763.605064:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517795712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827763.605070:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827763.605077:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827763.605079:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827763.605086:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 770 00000100:00080000:2.0:1787827763.605090:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827763.605098:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827763.605103:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827763.605121:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517795840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827763.605128:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827763.605136:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827763.605143:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827763.605157:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 770 00000100:00080000:2.0:1787827763.605168:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827763.605177:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827763.605188:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827763.605216:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517795968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827764.436765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827764.436833:0:14479:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 2 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787827764.436849:0:14479:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8b2e901ad800 lock: 00000000f95570e9/0x61d2feb329d09cca lrc: 2/0,0 mode: PR/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400000000 nid: local remote: 0xb0788cb3d22916b2 expref: -99 pid: 21372 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:1.0:1787827764.436851:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000080:00200000:2.0:1787827764.436886:0:14479:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x1e:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787827764.436903:0:14479:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x1e:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787827764.436932:0:14479:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-OST0001-osc-ffff8b2e901ad800 lock: 00000000634acb41/0x61d2feb329d0917f lrc: 2/0,0 mode: PW/PW res: [0x280000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x800028400020000 nid: local remote: 0xb0788cb3d228d37c expref: -99 pid: 18902 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00000080:00200000:2.0:1787827764.436949:0:14479:0:(vvp_io.c:1889:vvp_io_init()) [0x200000401:0x13:0x0] ignore/verify layout 1/0, layout version 0 restore needed 0 00000080:00200000:2.0:1787827764.436955:0:14479:0:(vvp_io.c:304:vvp_io_fini()) [0x200000401:0x13:0x0] ignore/verify layout 1/0, layout version 0 need write layout 0, restore needed 0, invalidate_lock 1 00010000:00010000:2.0:1787827764.437006:0:14479:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8b2e901ad800 lock: 00000000f95570e9/0x61d2feb329d09cca lrc: 1/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0xb0788cb3d22916b2 expref: -99 pid: 21372 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827764.437016:0:14479:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-OST0001-osc-ffff8b2e901ad800 lock: 00000000634acb41/0x61d2feb329d0917f lrc: 1/0,0 mode: --/PW res: [0x280000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400020020 nid: local remote: 0xb0788cb3d228d37c expref: -99 pid: 18902 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827764.437023:0:14479:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 2 locks packed 00010000:00010000:2.0:1787827764.437078:0:14479:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8b2e901ad800 lock: 00000000f95570e9/0x61d2feb329d09cca lrc: 0/0,0 mode: --/PR res: [0x280000400:0x8:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400000020 nid: local remote: 0xb0788cb3d22916b2 expref: -99 pid: 21372 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827764.437120:0:14479:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-OST0001-osc-ffff8b2e901ad800 lock: 00000000634acb41/0x61d2feb329d0917f lrc: 0/0,0 mode: --/PW res: [0x280000400:0x6:0x0].0x0 rrc: 2 type: EXT [0->18446744073709551615] (req 0->18446744073709551615) gid 0 flags: 0x804869400020020 nid: local remote: 0xb0788cb3d228d37c expref: -99 pid: 18902 timeout: 0 lvb_type: 1 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827764.437171:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827764.437176:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827764.437180:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827764.628751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827765.460757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827765.652741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827766.484823:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827766.676783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827767.508750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827767.508790:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827767.700764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827768.532746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:1.0:1787827768.533113:0:2348:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8b2e88043c00 x1874672517796224/t0(0) o103->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787827768.724760:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787827768.724760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827768.724770:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827768.724775:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827768.724789:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 775 00000100:00080000:1.0:1787827768.724803:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827768.724818:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827768.724866:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517796352/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827768.724894:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827768.724897:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827768.724899:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827768.724902:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 775 00000100:00080000:1.0:1787827768.724904:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827768.724907:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827768.724931:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827768.724945:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517796480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827768.724959:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827768.724968:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827768.724972:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827768.724978:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 775 00000100:00080000:1.0:1787827768.724984:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827768.724988:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827768.724993:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827768.725006:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517796608/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827768.725014:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827768.725017:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827768.725019:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827768.725022:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 775 00000100:00080000:1.0:1787827768.725026:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827768.725029:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827768.725033:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827768.725041:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517796736/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827768.725047:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827768.725049:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827768.725051:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827768.725056:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 775 00000100:00080000:1.0:1787827768.725061:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827768.725066:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827768.725069:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827768.725077:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517796864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827768.725086:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827768.725090:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827768.725093:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827768.725099:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 775 00000100:00080000:1.0:1787827768.725104:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827768.725110:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827768.725121:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827768.725139:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517796992/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827768.725148:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827768.725150:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827768.725152:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827768.725155:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 775 00000100:00080000:1.0:1787827768.725162:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827768.725169:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827768.725174:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827768.725190:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517797120/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827769.556729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827769.556787:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827769.748791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827770.580761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827770.772793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827771.604741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827771.604798:0:6036:0:(ldlm_lockd.c:2801:ldlm_bl_get_work()) Got 1 locks of 0 total in blp. (0 blwis in pool) 00010000:00010000:2.0:1787827771.604812:0:6036:0:(ldlm_request.c:1391:ldlm_cli_cancel_local()) ### client-side cancel ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8c600/0x61d2feb329d09d17 lrc: 2/0,0 mode: PR/PR res: [0x200000401:0x20:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x8400000000 nid: local remote: 0xb0788cb3d2291887 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827771.604899:0:6036:0:(ldlm_request.c:1439:__ldlm_pack_lock()) ### packing ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8c600/0x61d2feb329d09d17 lrc: 1/0,0 mode: --/PR res: [0x200000401:0x20:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xb0788cb3d2291887 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827771.604911:0:6036:0:(ldlm_request.c:1498:ldlm_cancel_pack()) 1 locks packed 00010000:00010000:2.0:1787827771.604948:0:6036:0:(ldlm_lock.c:193:ldlm_lock_put()) ### final lock_put on destroyed lock, freeing it. ns: lustre-MDT0000-mdc-ffff8b2e84abe800 lock: ffff8b2e88b8c600/0x61d2feb329d09d17 lrc: 0/0,0 mode: --/PR res: [0x200000401:0x20:0x0].0x0 bits 0x11/0x0 rrc: 4 type: IBT gid 0 flags: 0x4809400000020 nid: local remote: 0xb0788cb3d2291887 expref: -99 pid: 21402 timeout: 0 lvb_type: 0 lru_score: 1 lru_type: 0 00010000:00010000:2.0:1787827771.604968:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827771.604969:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827771.604971:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827771.604985:0:2349:0:(ptlrpcd.c:344:ptlrpcd_check()) transfer 1 async RPCs [1->0] 00000100:00080000:2.0:1787827771.605007:0:2349:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8b2e8801e1c0 x1874672517797248/t0(0) o103->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:17/18 lens 328/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787827771.796778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827772.628769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827772.628820:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827772.820812:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827773.652746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:1.0:1787827773.844757:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827773.844766:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827773.844772:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787827773.844773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827773.844788:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 780 00000100:00080000:1.0:1787827773.844802:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827773.844811:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827773.844863:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517797376/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827773.844894:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827773.844898:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827773.844901:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827773.844923:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 780 00000100:00080000:1.0:1787827773.844928:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827773.844934:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827773.844949:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827773.844970:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517797504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827773.844984:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827773.844989:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827773.844992:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827773.844996:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 780 00000100:00080000:1.0:1787827773.845001:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827773.845006:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827773.845012:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827773.845027:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517797632/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827773.845047:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827773.845052:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827773.845054:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827773.845061:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 780 00000100:00080000:1.0:1787827773.845066:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827773.845071:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827773.845077:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827773.845095:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517797760/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827773.845116:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827773.845124:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827773.845129:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827773.845139:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 780 00000100:00080000:1.0:1787827773.845147:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827773.845159:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827773.845167:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827773.845183:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517797888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827773.845199:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827773.845208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827773.845213:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827773.845238:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 780 00000100:00080000:1.0:1787827773.845246:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827773.845257:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827773.845268:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827773.845290:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517798016/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827773.845305:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827773.845313:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827773.845333:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827773.845339:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 780 00000100:00080000:1.0:1787827773.845347:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827773.845357:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827773.845365:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827773.845385:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517798144/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827774.676767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827774.676840:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827774.868731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827775.700764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827775.700838:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827775.892777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827776.724750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827776.724863:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827776.725052:0:2349:0:(client.c:1845:ptlrpc_send_new_req()) @@@ req waiting for recovery: (FULL != DISCONN) req@ffff8b2e88038780 x1874672517798272/t0(0) o103->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:17/18 lens 336/224 e 0 to 0 dl 0 ref 1 fl Rpc:WQU/200/ffffffff rc 0/-1 job:'ldlm_bl.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787827776.916769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827777.748713:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827777.748798:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827777.940791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827778.772751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827778.772853:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827778.964742:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827778.964754:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827778.964775:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827778.964786:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 785 00000100:00100000:2.0:1787827778.964795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827778.964798:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827778.964815:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827778.964850:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517798400/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827778.964873:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827778.964875:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827778.964877:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827778.964879:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 785 00000100:00080000:1.0:1787827778.964882:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827778.964885:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827778.964894:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827778.964905:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517798528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827778.964918:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827778.964920:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827778.964921:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827778.964924:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 785 00000100:00080000:1.0:1787827778.964925:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827778.964929:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827778.964932:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827778.964941:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517798656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827778.964947:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827778.964950:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827778.964951:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827778.964953:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 785 00000100:00080000:1.0:1787827778.964956:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827778.964958:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827778.964960:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827778.964968:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517798784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827778.964973:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827778.964975:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827778.964977:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827778.964984:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 785 00000100:00080000:1.0:1787827778.964989:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827778.964992:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827778.964995:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827778.965002:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517798912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827778.965007:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827778.965010:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827778.965012:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827778.965015:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 785 00000100:00080000:1.0:1787827778.965017:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827778.965020:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827778.965022:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827778.965030:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517799040/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827778.965035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827778.965037:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827778.965038:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827778.965043:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 785 00000100:00080000:1.0:1787827778.965046:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827778.965051:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827778.965057:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827778.965069:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517799168/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827779.796725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827779.988737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827780.820775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827780.820853:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827781.012764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827781.844723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827781.844831:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827782.036761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827782.868751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827783.892764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827783.892841:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787827784.084740:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827784.084747:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827784.084753:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827784.084763:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 790 00000100:00080000:1.0:1787827784.084772:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827784.084779:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827784.084811:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517799296/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827784.084837:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827784.084840:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827784.084841:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827784.084845:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 790 00000100:00080000:1.0:1787827784.084847:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827784.084850:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827784.084859:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827784.084871:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517799424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827784.084881:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827784.084884:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827784.084885:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827784.084888:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 790 00000100:00080000:1.0:1787827784.084891:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827784.084893:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827784.084896:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827784.084904:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517799552/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827784.084911:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827784.084913:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827784.084915:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827784.084917:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 790 00000100:00080000:1.0:1787827784.084920:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827784.084921:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827784.084924:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827784.084935:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517799680/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827784.084942:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827784.084945:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827784.084947:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827784.084949:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 790 00000100:00080000:1.0:1787827784.084952:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827784.084954:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827784.084957:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827784.084967:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517799808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827784.084977:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827784.084979:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827784.084980:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827784.084986:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 790 00000100:00080000:1.0:1787827784.084991:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827784.084995:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827784.084999:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827784.085007:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517799936/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827784.085022:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827784.085027:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827784.085030:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827784.085038:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 790 00000100:00080000:1.0:1787827784.085044:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827784.085048:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827784.085050:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827784.085057:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517800064/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827784.916771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827784.916806:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827785.940736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827786.964810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827786.964872:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827787.988828:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827787.988864:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827789.012777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827790.036780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827791.060719:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827792.084737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827793.108719:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827794.132771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827794.324786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827795.156762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827795.348814:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827796.180730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827796.372841:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827797.204765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827797.397038:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827798.228797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827798.420814:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827799.252712:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827799.444822:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827800.276768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827800.468874:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827801.492815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827802.324838:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827802.516753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827803.348808:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827803.540750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827804.564745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827804.564799:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827804.564807:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827804.564812:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827804.564822:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 810 00000100:00080000:2.0:1787827804.564836:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827804.564844:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827804.564892:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517802880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827804.564921:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827804.564935:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827804.564936:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827804.564941:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 810 00000100:00080000:2.0:1787827804.564944:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827804.564947:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827804.564957:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827804.564969:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517803008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827804.564978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827804.564981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827804.564982:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827804.564984:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 810 00000100:00080000:2.0:1787827804.564986:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827804.564988:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827804.565001:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827804.565009:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517803136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827804.565016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827804.565019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827804.565020:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827804.565023:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 810 00000100:00080000:2.0:1787827804.565026:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827804.565029:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827804.565033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827804.565039:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517803264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827804.565044:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827804.565047:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827804.565048:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827804.565052:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 810 00000100:00080000:2.0:1787827804.565054:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827804.565058:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827804.565062:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827804.565071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517803392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827804.565078:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827804.565081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827804.565083:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827804.565086:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 810 00000100:00080000:2.0:1787827804.565088:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827804.565091:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827804.565095:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827804.565106:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517803520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827804.565117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827804.565121:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827804.565123:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827804.565128:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 810 00000100:00080000:2.0:1787827804.565135:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827804.565146:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827804.565152:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827804.565168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517803648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827805.396915:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827805.588774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827806.420983:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827806.612771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827807.636846:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827808.468987:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827808.660788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827809.492958:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827809.684812:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827809.684869:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827809.684884:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827809.684894:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827809.684915:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 816 00000100:00080000:2.0:1787827809.684937:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827809.684950:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827809.685029:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517803776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827809.685076:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827809.685084:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827809.685088:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827809.685097:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 816 00000100:00080000:2.0:1787827809.685108:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827809.685119:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827809.685140:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827809.685180:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517803904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827809.685226:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827809.685236:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827809.685242:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827809.685261:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 816 00000100:00080000:2.0:1787827809.685273:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827809.685284:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827809.685297:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827809.685321:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517804032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827809.685362:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827809.685376:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827809.685384:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827809.685407:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 816 00000100:00080000:2.0:1787827809.685419:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827809.685429:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827809.685439:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827809.685470:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517804160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827809.685513:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827809.685520:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827809.685524:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827809.685531:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 816 00000100:00080000:2.0:1787827809.685537:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827809.685549:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827809.685557:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827809.685577:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517804288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827809.685611:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827809.685625:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827809.685634:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827809.685685:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 816 00000100:00080000:2.0:1787827809.685693:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827809.685714:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827809.685734:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827809.685768:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517804416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827809.685783:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827809.685801:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827809.685806:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827809.685815:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 816 00000100:00080000:2.0:1787827809.685823:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827809.685840:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827809.685852:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827809.685903:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517804544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827810.516809:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827810.708772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827811.540904:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827811.732805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827812.564813:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827812.756697:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827813.780832:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827814.612882:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827814.804781:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787827814.804792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827814.804797:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827814.804810:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827814.804831:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 821 00000100:00080000:2.0:1787827814.804850:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827814.804862:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827814.804929:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517804672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827814.804971:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827814.804977:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827814.804980:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827814.804985:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 821 00000100:00080000:2.0:1787827814.804990:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827814.804996:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827814.805011:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827814.805028:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517804800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827814.805044:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827814.805050:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827814.805057:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827814.805086:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 821 00000100:00080000:2.0:1787827814.805092:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827814.805097:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827814.805105:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827814.805119:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517804928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827814.805132:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827814.805137:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827814.805146:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827814.805152:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 821 00000100:00080000:2.0:1787827814.805157:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827814.805162:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827814.805168:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827814.805181:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517805056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827814.805194:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827814.805205:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827814.805213:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827814.805220:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 821 00000100:00080000:2.0:1787827814.805226:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827814.805231:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827814.805238:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827814.805253:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517805184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827814.805266:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827814.805271:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827814.805273:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827814.805280:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 821 00000100:00080000:2.0:1787827814.805286:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827814.805295:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827814.805327:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827814.805351:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517805312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827814.805384:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827814.805396:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827814.805399:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827814.805407:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 821 00000100:00080000:2.0:1787827814.805414:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827814.805426:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827814.805436:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827814.805455:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517805440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827815.636936:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827815.828845:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827816.852803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827817.684857:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827817.876818:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827818.708948:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827818.900799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827819.732892:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827819.924794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827819.924826:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827819.924841:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827819.924850:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827819.924875:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 826 00000100:00080000:2.0:1787827819.924903:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827819.924923:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827819.924998:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517805568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827819.925065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827819.925072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827819.925076:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827819.925085:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 826 00000100:00080000:2.0:1787827819.925093:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827819.925102:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827819.925122:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827819.925157:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517805696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827819.925182:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827819.925190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827819.925194:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827819.925205:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 826 00000100:00080000:2.0:1787827819.925214:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827819.925223:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827819.925236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827819.925321:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517805824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827819.925388:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827819.925396:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827819.925399:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827819.925414:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 826 00000100:00080000:2.0:1787827819.925435:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827819.925447:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827819.925462:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827819.925485:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517805952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827819.925511:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827819.925515:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827819.925518:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827819.925523:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 826 00000100:00080000:2.0:1787827819.925529:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827819.925532:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827819.925537:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827819.925552:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517806080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827819.925576:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827819.925580:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827819.925581:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827819.925585:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 826 00000100:00080000:2.0:1787827819.925594:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827819.925598:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827819.925604:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827819.925639:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517806208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827819.925653:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827819.925658:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827819.925663:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827819.925714:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 826 00000100:00080000:2.0:1787827819.925723:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827819.925736:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827819.925759:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827819.925781:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517806336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827820.756885:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827820.948764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827821.780875:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827821.972825:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827822.804910:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827822.996840:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827823.828895:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827824.020815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827824.852893:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827825.044791:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827825.044820:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827825.044830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827825.044838:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827825.044854:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 831 00000100:00080000:2.0:1787827825.044873:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827825.044886:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827825.044947:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517806464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827825.044997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827825.045002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827825.045005:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827825.045011:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 831 00000100:00080000:2.0:1787827825.045015:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827825.045020:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827825.045039:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827825.045070:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517806592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827825.045116:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827825.045125:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827825.045130:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827825.045137:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 831 00000100:00080000:2.0:1787827825.045144:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827825.045151:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827825.045161:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827825.045185:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517806720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827825.045201:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827825.045206:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827825.045209:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827825.045214:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 831 00000100:00080000:2.0:1787827825.045219:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827825.045224:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827825.045231:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827825.045292:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517806848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827825.045319:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827825.045333:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827825.045349:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827825.045358:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 831 00000100:00080000:2.0:1787827825.045365:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827825.045372:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827825.045382:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827825.045407:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517806976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827825.045422:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827825.045431:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827825.045436:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827825.045463:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 831 00000100:00080000:2.0:1787827825.045471:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827825.045476:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827825.045482:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827825.045494:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517807104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827825.045523:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827825.045531:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827825.045533:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827825.045541:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 831 00000100:00080000:2.0:1787827825.045546:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827825.045550:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827825.045557:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827825.045571:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517807232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787827825.876831:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827826.068820:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827826.900748:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827827.092735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827827.924975:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827828.116731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827828.948897:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827829.140735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827829.972854:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827830.164731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827830.164769:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827830.164782:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827830.164788:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827830.164803:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 836 00000100:00080000:2.0:1787827830.164828:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827830.164838:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827830.164888:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517807360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827830.164917:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827830.164920:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827830.164921:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827830.164925:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 836 00000100:00080000:2.0:1787827830.164930:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827830.164935:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827830.164946:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827830.164962:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517807488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827830.164972:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827830.164977:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827830.164979:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827830.164986:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 836 00000100:00080000:2.0:1787827830.164991:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827830.164995:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827830.165002:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827830.165017:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517807616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827830.165042:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827830.165047:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827830.165049:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827830.165055:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 836 00000100:00080000:2.0:1787827830.165061:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827830.165068:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827830.165074:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827830.165093:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517807744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827830.165122:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827830.165133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827830.165142:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827830.165153:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 836 00000100:00080000:2.0:1787827830.165165:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827830.165179:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827830.165187:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827830.165264:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517807872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827830.165295:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827830.165303:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827830.165306:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827830.165312:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 836 00000100:00080000:2.0:1787827830.165319:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827830.165327:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827830.165334:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827830.165350:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517808000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827830.165359:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827830.165361:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827830.165364:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827830.165366:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 836 00000100:00080000:2.0:1787827830.165372:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827830.165380:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827830.165386:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827830.165399:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517808128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827830.996836:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827831.188834:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827832.021013:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827833.044819:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827834.068829:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827835.092836:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827835.284728:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827835.284736:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827835.284740:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827835.284749:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 841 00000100:00080000:2.0:1787827835.284762:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827835.284770:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827835.284807:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517808256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827835.284851:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827835.284858:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827835.284861:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827835.284868:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 841 00000100:00080000:2.0:1787827835.284874:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827835.284879:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827835.284891:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827835.284914:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517808384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827835.284933:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827835.284937:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827835.284938:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827835.284942:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 841 00000100:00080000:2.0:1787827835.284945:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827835.284948:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827835.284953:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827835.284971:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517808512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827835.285003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827835.285006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827835.285008:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827835.285012:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 841 00000100:00080000:2.0:1787827835.285018:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827835.285027:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827835.285033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827835.285046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517808640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827835.285071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827835.285075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827835.285078:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827835.285084:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 841 00000100:00080000:2.0:1787827835.285094:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827835.285105:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827835.285115:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827835.285136:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517808768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827835.285186:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827835.285210:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827835.285214:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827835.285219:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 841 00000100:00080000:2.0:1787827835.285224:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827835.285231:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827835.285238:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827835.285249:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517808896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827835.285276:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827835.285279:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827835.285282:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827835.285287:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 841 00000100:00080000:2.0:1787827835.285290:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827835.285299:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827835.285307:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827835.285321:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517809024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827836.116789:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827837.140832:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827837.332779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827838.164859:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827838.356719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827839.188824:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827839.380746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827840.212799:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827840.404733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827841.236794:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827841.428749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827842.260816:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827842.452717:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827843.284777:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827843.284811:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827844.308723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827844.308807:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827845.332726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827845.332779:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827845.524760:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827845.524767:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827845.524770:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827845.524779:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 851 00000100:00080000:2.0:1787827845.524788:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827845.524794:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827845.524828:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517810048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827845.524849:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827845.524852:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827845.524853:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827845.524857:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 851 00000100:00080000:2.0:1787827845.524860:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827845.524864:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827845.524872:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827845.524888:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517810176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827845.524902:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827845.524906:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827845.524909:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827845.524914:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 851 00000100:00080000:2.0:1787827845.524918:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827845.524923:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827845.524930:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827845.524948:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517810304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827845.524957:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827845.524960:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827845.524962:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827845.524965:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 851 00000100:00080000:2.0:1787827845.524969:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827845.524972:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827845.524979:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827845.524995:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517810432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827845.525014:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827845.525019:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827845.525021:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827845.525031:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 851 00000100:00080000:2.0:1787827845.525035:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827845.525042:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827845.525046:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827845.525054:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517810560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827845.525065:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827845.525072:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827845.525074:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827845.525077:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 851 00000100:00080000:2.0:1787827845.525080:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827845.525082:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827845.525087:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827845.525118:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517810688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827845.525130:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827845.525133:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827845.525134:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827845.525137:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 851 00000100:00080000:2.0:1787827845.525141:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827845.525147:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827845.525156:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827845.525169:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517810816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827846.356771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827846.356891:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827847.380752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787827847.380922:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827848.404724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827849.428719:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827849.428768:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827850.452740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827850.452794:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827851.476765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787827851.668790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827852.500774:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827852.692758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827853.524764:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827853.716743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827854.548812:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827854.740728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827855.572752:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827855.764736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827856.596806:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827856.788730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827857.620862:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827857.812800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827858.644769:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827858.836800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827859.668913:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827859.860808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827860.692795:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827860.884772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827861.716779:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827861.908868:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827862.740823:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827862.932909:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827863.764908:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827863.956847:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827863.957155:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827864.788884:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827864.980783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827865.812822:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827866.004777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827866.004828:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827866.836846:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827867.028740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827867.028847:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827867.860797:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827868.052724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827868.052726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827868.884899:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827869.076738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827869.076743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827869.908937:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827870.100822:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827870.100823:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827870.932885:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827871.124779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827871.124838:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827871.956857:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827872.148817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827872.148817:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827872.980869:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827873.172746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827873.172801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827874.004833:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827874.196757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827874.196760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827875.028866:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827875.220748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827875.220809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787827876.052811:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827876.244785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827876.244792:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827876.244806:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827876.244815:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827876.244837:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 882 00000100:00080000:1.0:1787827876.244856:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827876.244869:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827876.244988:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517815424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827876.245038:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827876.245044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827876.245046:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827876.245052:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 882 00000100:00080000:1.0:1787827876.245058:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827876.245065:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827876.245080:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827876.245116:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517815552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827876.245125:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827876.245128:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827876.245130:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827876.245135:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 882 00000100:00080000:1.0:1787827876.245139:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827876.245151:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827876.245157:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827876.245167:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517815680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827876.245174:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827876.245177:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827876.245178:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827876.245180:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 882 00000100:00080000:1.0:1787827876.245182:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827876.245185:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827876.245188:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827876.245219:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517815808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827876.245231:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827876.245235:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827876.245238:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827876.245243:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 882 00000100:00080000:1.0:1787827876.245250:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827876.245255:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827876.245260:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827876.245277:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517815936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827876.245292:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827876.245296:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827876.245299:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827876.245304:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 882 00000100:00080000:1.0:1787827876.245310:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827876.245315:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827876.245321:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827876.245335:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517816064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827876.245347:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827876.245350:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827876.245352:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827876.245359:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 882 00000100:00080000:1.0:1787827876.245365:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827876.245390:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827876.245393:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827876.245405:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517816192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827877.076831:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827877.268744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827878.292817:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827879.124905:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827879.316760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827880.148935:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827880.340743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827881.364734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827882.196790:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827882.388749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827883.220815:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827883.412730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827884.436744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827885.268792:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827885.460708:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827886.292849:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827886.484777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827887.508768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827888.340836:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827888.532757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827889.364807:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827889.556769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827890.580747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827891.412816:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827891.604754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827892.436811:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827892.628787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827893.652798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827894.484795:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827894.676786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827895.508803:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827895.700765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827896.532786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827896.724783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827897.556773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827897.556838:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827897.748773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827898.580777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827898.580830:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827898.772763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827899.604786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827899.796787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827900.628805:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827900.628806:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827900.820760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827901.652799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827901.652856:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827901.844789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827902.676751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827902.868751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827903.700773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827903.700832:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827903.892741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827904.724790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827904.724821:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827904.916778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827905.748793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827905.940789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827906.772742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827906.772831:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827906.964761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827907.796740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827907.796816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827907.988805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827908.820730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787827909.012729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827909.844776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827909.844819:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827910.036738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827910.868715:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787827910.868770:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827911.060771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827912.084788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827912.916833:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827913.108750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827913.940761:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827914.132732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827915.156772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827915.988798:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827916.180744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827916.180778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827917.012813:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827917.204773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787827917.204833:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827918.228741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827919.060795:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827919.252711:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827920.084797:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827920.276711:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827921.300738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827922.132838:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827922.324709:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827923.156894:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827923.348782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827924.372699:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827925.204939:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827925.396806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827926.228929:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827926.420839:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827927.444803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827928.277062:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827928.468759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827929.300984:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827929.492765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827930.516804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:1.0:1787827930.517128:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787827930.517185:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787827930.517206:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827930.517266:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827930.517275:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827930.517284:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827930.517293:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827930.517303:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787827930.517311:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00010000:00010000:1.0:1787827931.348832:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827931.540774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827932.372800:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827932.564756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787827932.564819:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827932.564825:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827932.564829:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827932.564839:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 938 00000100:00080000:2.0:1787827932.564849:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827932.564856:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827932.564893:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517825280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827932.564919:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827932.564923:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827932.564925:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827932.564930:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 938 00000100:00080000:2.0:1787827932.564935:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827932.564940:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827932.564953:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827932.564971:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517825408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827932.564983:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827932.564986:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827932.564988:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827932.564992:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 938 00000100:00080000:2.0:1787827932.564996:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827932.565000:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827932.565007:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827932.565020:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517825536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827932.565032:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827932.565035:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827932.565039:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827932.565043:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 938 00000100:00080000:2.0:1787827932.565047:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827932.565051:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827932.565056:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827932.565072:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517825664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827932.565083:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827932.565087:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827932.565088:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827932.565094:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 938 00000100:00080000:2.0:1787827932.565099:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827932.565103:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827932.565109:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827932.565123:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517825792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827932.565137:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827932.565146:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827932.565151:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827932.565158:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 938 00000100:00080000:2.0:1787827932.565162:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827932.565171:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827932.565188:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827932.565208:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517825920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827932.565225:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827932.565229:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827932.565231:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827932.565236:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 938 00000100:00080000:2.0:1787827932.565240:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827932.565246:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827932.565252:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827932.565270:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517826048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787827933.588801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787827934.420939:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827934.612873:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827935.444870:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827935.636867:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827936.468768:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827936.660727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827937.684714:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827937.684755:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.684764:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827937.684768:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.684778:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 944 00000100:00080000:1.0:1787827937.684791:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.684798:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827937.684842:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517826176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827937.684870:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.684874:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827937.684876:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.684880:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 944 00000100:00080000:1.0:1787827937.684883:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.684886:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827937.684896:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827937.684907:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517826304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827937.684937:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.684940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827937.684943:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.684947:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 944 00000100:00080000:1.0:1787827937.684949:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.684951:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827937.684955:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827937.684964:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517826432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827937.684970:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.684972:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827937.684973:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.684975:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 944 00000100:00080000:1.0:1787827937.684980:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.684983:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827937.684988:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827937.685005:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517826560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827937.685016:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.685019:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827937.685020:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.685024:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 944 00000100:00080000:1.0:1787827937.685027:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.685029:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827937.685033:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827937.685043:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517826688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827937.685056:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.685058:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827937.685059:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.685067:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 944 00000100:00080000:1.0:1787827937.685074:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.685083:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827937.685089:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827937.685101:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517826816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827937.685158:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827937.685161:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827937.685163:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827937.685167:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 944 00000100:00080000:1.0:1787827937.685178:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827937.685186:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827937.685189:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827937.685196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517826944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827938.516845:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827938.708794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827939.540911:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827939.732739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787827940.756748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827941.588837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827941.780783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827942.612898:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827942.804743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827942.804755:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827942.804762:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827942.804767:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827942.804779:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 949 00000100:00080000:1.0:1787827942.804793:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827942.804802:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827942.804850:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517827072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827942.804878:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827942.804881:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827942.804882:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827942.804886:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 949 00000100:00080000:1.0:1787827942.804888:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827942.804891:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827942.804901:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827942.804911:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517827200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827942.804933:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827942.804936:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827942.804937:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827942.804941:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 949 00000100:00080000:1.0:1787827942.804942:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827942.804945:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827942.804948:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827942.804958:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517827328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827942.804965:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827942.804967:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827942.804968:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827942.804971:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 949 00000100:00080000:1.0:1787827942.804994:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827942.804998:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827942.805001:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827942.805011:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517827456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827942.805019:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827942.805023:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827942.805024:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827942.805027:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 949 00000100:00080000:1.0:1787827942.805032:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827942.805035:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827942.805037:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827942.805044:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517827584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827942.805054:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827942.805058:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827942.805060:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827942.805066:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 949 00000100:00080000:1.0:1787827942.805071:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827942.805082:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827942.805091:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827942.805107:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517827712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827942.805129:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827942.805135:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827942.805138:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827942.805142:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 949 00000100:00080000:1.0:1787827942.805145:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827942.805148:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827942.805153:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827942.805162:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517827840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787827943.828713:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827944.660831:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827944.852711:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827945.684798:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827945.876733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827946.708763:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827946.900744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827947.732901:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827947.924764:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827947.924774:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827947.924779:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827947.924791:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 954 00000100:00080000:2.0:1787827947.924805:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827947.924814:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827947.924872:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517827968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827947.924918:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827947.924925:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827947.924930:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827947.924938:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 954 00000100:00080000:2.0:1787827947.924945:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827947.924951:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827947.924964:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827947.924986:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517828096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827947.925007:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827947.925013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827947.925016:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827947.925024:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 954 00000100:00080000:2.0:1787827947.925030:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827947.925038:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827947.925048:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827947.925069:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517828224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827947.925089:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827947.925095:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827947.925099:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827947.925106:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 954 00000100:00080000:2.0:1787827947.925112:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827947.925118:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827947.925128:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827947.925149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517828352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827947.925168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827947.925174:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827947.925178:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827947.925186:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 954 00000100:00080000:2.0:1787827947.925193:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827947.925200:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827947.925209:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827947.925231:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517828480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827947.925251:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827947.925258:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827947.925264:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827947.925280:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 954 00000100:00080000:2.0:1787827947.925288:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827947.925297:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827947.925306:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827947.925329:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517828608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827947.925355:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827947.925362:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827947.925367:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827947.925373:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 954 00000100:00080000:2.0:1787827947.925402:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827947.925410:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827947.925418:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827947.925438:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517828736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827948.756875:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827949.780811:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827950.804783:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827951.828958:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827952.852956:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827953.044792:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827953.044805:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827953.044812:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827953.044826:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 959 00000100:00080000:2.0:1787827953.044840:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827953.044861:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827953.044922:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517828864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827953.044950:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827953.044954:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827953.044957:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827953.044962:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 959 00000100:00080000:2.0:1787827953.044966:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827953.044990:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827953.045003:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827953.045025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517828992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827953.045050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827953.045054:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827953.045057:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827953.045062:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 959 00000100:00080000:2.0:1787827953.045066:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827953.045070:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827953.045076:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827953.045090:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517829120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827953.045101:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827953.045104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827953.045105:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827953.045110:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 959 00000100:00080000:2.0:1787827953.045115:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827953.045118:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827953.045124:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827953.045144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517829248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827953.045163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827953.045168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827953.045170:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827953.045175:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 959 00000100:00080000:2.0:1787827953.045179:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827953.045183:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827953.045191:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827953.045206:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517829376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827953.045236:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827953.045246:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827953.045250:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827953.045259:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 959 00000100:00080000:2.0:1787827953.045265:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827953.045278:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827953.045291:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827953.045310:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517829504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827953.045340:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827953.045368:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827953.045371:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827953.045381:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 959 00000100:00080000:2.0:1787827953.045410:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827953.045417:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827953.045423:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827953.045443:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517829632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827953.876900:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827954.900917:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827955.924855:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827956.116739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827956.948795:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827957.140719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827957.972920:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827958.164741:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827958.164756:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827958.164761:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827958.164770:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 964 00000100:00080000:2.0:1787827958.164783:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827958.164792:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827958.164829:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517829760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827958.164862:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827958.164865:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827958.164867:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827958.164870:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 964 00000100:00080000:2.0:1787827958.164873:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827958.164877:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827958.164886:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827958.164898:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517829888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827958.164927:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827958.164930:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827958.164931:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827958.164935:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 964 00000100:00080000:2.0:1787827958.164938:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827958.164942:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827958.164946:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827958.164960:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517830016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827958.164967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827958.164971:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827958.164972:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827958.164975:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 964 00000100:00080000:2.0:1787827958.164978:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827958.164982:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827958.164990:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827958.165003:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517830144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827958.165011:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827958.165013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827958.165015:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827958.165018:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 964 00000100:00080000:2.0:1787827958.165021:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827958.165025:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827958.165030:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827958.165043:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517830272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827958.165062:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827958.165067:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827958.165070:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827958.165081:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 964 00000100:00080000:2.0:1787827958.165088:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827958.165094:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827958.165104:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827958.165117:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517830400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827958.165144:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827958.165147:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827958.165149:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827958.165152:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 964 00000100:00080000:2.0:1787827958.165155:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827958.165158:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827958.165161:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827958.165172:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517830528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827958.996777:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827959.188756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827960.020970:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827960.212869:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827961.044901:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:2.0:1787827962.068916:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827962.260810:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827963.092977:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787827963.284824:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827963.284841:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787827963.284849:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787827963.284871:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787827963.284871:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 969 00000100:00080000:2.0:1787827963.284899:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827963.284915:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787827963.284968:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517830656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827963.285007:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827963.285012:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827963.285015:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827963.285021:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 969 00000100:00080000:2.0:1787827963.285027:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827963.285032:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827963.285045:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827963.285072:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517830784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827963.285110:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827963.285119:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827963.285125:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827963.285137:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 969 00000100:00080000:2.0:1787827963.285147:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827963.285157:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827963.285168:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787827963.285196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517830912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827963.285245:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827963.285250:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827963.285253:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827963.285263:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 969 00000100:00080000:2.0:1787827963.285270:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827963.285300:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827963.285309:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827963.285327:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517831040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827963.285377:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827963.285401:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787827963.285407:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827963.285428:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 969 00000100:00080000:2.0:1787827963.285449:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827963.285463:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827963.285469:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827963.285491:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517831168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827963.285511:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827963.285526:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787827963.285531:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827963.285545:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 969 00000100:00080000:2.0:1787827963.285554:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827963.285559:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827963.285566:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827963.285581:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517831296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787827963.285615:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787827963.285622:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787827963.285627:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787827963.285635:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 969 00000100:00080000:2.0:1787827963.285642:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787827963.285698:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787827963.285706:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787827963.285732:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517831424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827964.116953:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827964.308732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827965.140921:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827965.332809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827966.165029:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827966.356810:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827966.356898:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827967.189047:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827967.380817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827967.380877:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827968.212844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827968.404744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827968.404836:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827969.236940:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827969.428819:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827969.428879:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827970.260995:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827970.452793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827970.452880:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827971.284923:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827971.476854:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827971.476861:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827972.308953:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827972.500777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827973.332877:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827973.524804:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827973.524926:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827974.356884:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827974.548730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827974.548803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827975.380879:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827975.572749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827975.572782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827976.404821:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827976.596773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827976.596780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827977.428868:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827977.620764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827977.620764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827978.452951:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827978.644788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827978.644835:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827979.476940:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827979.668725:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827979.668863:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827980.500819:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827980.692737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827981.524908:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827981.716806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827982.548838:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827982.740757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827983.572993:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827983.764846:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827983.764885:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827984.596929:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827984.788756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827985.620789:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827985.812723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827986.644841:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787827986.836745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787827986.836767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827987.668781:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827987.860711:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787827987.860714:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787827988.692800:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827988.884760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827988.884780:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827988.884790:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827988.884797:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827988.884810:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 995 00000100:00080000:1.0:1787827988.884826:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827988.884840:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827988.884898:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517835136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827988.884944:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827988.884949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827988.884952:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827988.884957:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 995 00000100:00080000:1.0:1787827988.884961:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827988.884965:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827988.884979:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827988.884994:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517835264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827988.885009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827988.885016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827988.885019:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827988.885027:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 995 00000100:00080000:1.0:1787827988.885050:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827988.885055:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827988.885059:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827988.885067:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517835392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827988.885074:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827988.885077:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827988.885078:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827988.885081:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 995 00000100:00080000:1.0:1787827988.885084:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827988.885087:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827988.885090:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827988.885114:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517835520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827988.885121:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827988.885124:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827988.885125:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827988.885129:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 995 00000100:00080000:1.0:1787827988.885132:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827988.885135:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827988.885139:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827988.885149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517835648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827988.885186:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827988.885192:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827988.885195:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827988.885202:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 995 00000100:00080000:1.0:1787827988.885207:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827988.885212:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827988.885217:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827988.885232:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517835776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827988.885243:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827988.885247:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827988.885250:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827988.885254:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 995 00000100:00080000:1.0:1787827988.885259:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827988.885268:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827988.885277:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827988.885296:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517835904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827989.716913:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827989.908762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827990.740901:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827990.932757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827991.764854:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827991.956768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827992.788841:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827992.980730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827993.812815:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827994.004732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827994.004744:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827994.004754:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827994.004759:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827994.004773:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1000 00000100:00080000:1.0:1787827994.004789:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827994.004798:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827994.004844:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517836032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827994.004891:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827994.004895:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827994.004898:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827994.004902:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1000 00000100:00080000:1.0:1787827994.004907:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827994.004913:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827994.004927:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827994.004947:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517836160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827994.004962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827994.004965:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827994.004967:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827994.004972:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1000 00000100:00080000:1.0:1787827994.004975:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827994.004979:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827994.004984:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827994.005000:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517836288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827994.005010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827994.005013:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827994.005015:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827994.005019:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1000 00000100:00080000:1.0:1787827994.005023:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827994.005026:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827994.005034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827994.005054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517836416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827994.005104:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827994.005112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827994.005119:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827994.005128:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1000 00000100:00080000:1.0:1787827994.005135:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827994.005141:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827994.005149:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827994.005189:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517836544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827994.005208:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827994.005213:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827994.005216:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827994.005224:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1000 00000100:00080000:1.0:1787827994.005233:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827994.005244:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827994.005253:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827994.005273:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517836672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827994.005306:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827994.005312:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827994.005316:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827994.005328:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1000 00000100:00080000:1.0:1787827994.005337:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827994.005362:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827994.005371:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827994.005388:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517836800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827994.836860:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827995.028782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827995.860850:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827996.052814:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827996.885027:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827997.076808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:2.0:1787827997.077252:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787827997.077298:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787827997.077313:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827997.077370:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827997.077384:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827997.077395:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827997.077401:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827997.077408:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787827997.077414:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00010000:00010000:2.0:1787827997.908980:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827998.100809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787827998.932952:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787827999.124729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787827999.124732:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827999.124742:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787827999.124747:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827999.124757:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1005 00000100:00080000:1.0:1787827999.124770:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827999.124779:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787827999.124818:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517836928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827999.124873:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827999.124881:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827999.124884:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827999.124892:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1005 00000100:00080000:1.0:1787827999.124899:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827999.124908:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827999.124940:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827999.124977:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517837056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827999.124993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827999.124997:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827999.125000:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827999.125006:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1005 00000100:00080000:1.0:1787827999.125014:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827999.125020:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827999.125049:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787827999.125081:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517837184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827999.125098:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827999.125103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827999.125106:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827999.125113:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1005 00000100:00080000:1.0:1787827999.125120:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827999.125132:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827999.125143:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827999.125164:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517837312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827999.125204:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827999.125211:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787827999.125217:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827999.125229:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1005 00000100:00080000:1.0:1787827999.125245:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827999.125256:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827999.125264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827999.125286:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517837440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827999.125311:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827999.125319:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787827999.125323:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827999.125333:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1005 00000100:00080000:1.0:1787827999.125340:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827999.125349:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827999.125361:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827999.125390:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517837568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787827999.125445:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787827999.125456:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787827999.125458:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787827999.125465:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1005 00000100:00080000:1.0:1787827999.125472:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787827999.125479:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787827999.125487:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787827999.125512:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517837696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787827999.956975:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828000.148852:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828000.980811:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828001.172753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828002.196900:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828003.028886:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828003.220839:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828004.052828:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828004.244756:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.244767:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828004.244773:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828004.244782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828004.244785:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1010 00000100:00080000:1.0:1787828004.244802:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.244811:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828004.244855:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517837824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.244886:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.244890:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828004.244893:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828004.244897:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1010 00000100:00080000:1.0:1787828004.244901:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.244905:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828004.244917:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828004.244933:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517837952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.244951:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.244955:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828004.244957:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828004.244962:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1010 00000100:00080000:1.0:1787828004.244966:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.244969:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828004.244974:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828004.245005:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517838080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.245016:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.245038:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828004.245041:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828004.245046:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1010 00000100:00080000:1.0:1787828004.245056:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.245070:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828004.245079:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828004.245095:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517838208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.245108:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.245113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828004.245115:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828004.245124:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1010 00000100:00080000:1.0:1787828004.245128:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.245140:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828004.245152:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828004.245178:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517838336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.245243:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.245253:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828004.245275:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828004.245283:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1010 00000100:00080000:1.0:1787828004.245289:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.245307:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828004.245312:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828004.245328:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517838464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828004.245344:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828004.245347:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828004.245349:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828004.245352:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1010 00000100:00080000:1.0:1787828004.245356:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828004.245360:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828004.245365:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828004.245380:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517838592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828005.076889:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828005.268803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828006.100932:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828006.292807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828007.124863:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828007.316772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828007.316825:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828008.148897:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828008.340780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828008.340818:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828009.172921:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828009.364774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828009.364805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828010.196937:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828010.388753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828010.388753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828011.220813:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828011.412724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828011.412725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828012.244871:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828012.436750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828012.436806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828013.268883:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828013.460764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828013.460775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828014.292883:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828014.484763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828014.484802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828015.316939:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828015.508805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828015.508811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828016.340803:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828016.532711:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828016.532711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828017.364926:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828017.556791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828017.556828:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828018.388953:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828018.580775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828018.580829:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828019.412796:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828019.604709:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828019.604722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828020.436959:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828020.628793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828020.628798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828021.461010:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828021.652765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828021.652805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828022.484937:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828022.676756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828022.676779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828023.508851:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828023.701277:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828024.532888:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828024.724727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828025.556822:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828025.748828:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828026.580783:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828026.772777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828027.604989:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828027.796894:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828028.628923:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828028.820905:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828029.652960:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828029.844907:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828029.845024:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517842304:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828029.845161:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828029.845240:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff8b2e880392c0 x1874672517842304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828029.845277:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517842304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828029.845323:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828029.845332:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828029.845343:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517842304:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828029.845406:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517842432:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828029.845448:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828029.845468:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff8b2e88039a40 x1874672517842432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828029.845502:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517842432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828029.845512:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828029.845516:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828029.845520:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517842432:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828029.845526:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517842560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828029.845542:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828029.845550:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff8b2e8803b480 x1874672517842560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828029.845565:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517842560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828029.845572:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828029.845574:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828029.845576:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517842560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828029.845581:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517842688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828029.845593:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828029.845600:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff8b2e8803a1c0 x1874672517842688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828029.845618:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517842688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828029.845625:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828029.845628:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828029.845630:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517842688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828029.845635:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517842816:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828029.845671:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828029.845680:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff8b2e88039e00 x1874672517842816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828029.845690:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517842816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828029.845702:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828029.845704:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828029.845707:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517842816:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828029.845721:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517842944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828029.845736:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828029.845744:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff8b2e88039680 x1874672517842944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828029.845753:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517842944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828029.845760:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828029.845762:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828029.845764:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517842944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828029.845769:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517843072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828029.845780:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828029.845787:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828029/real 1787828029] req@ffff8b2e880383c0 x1874672517843072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828029.845796:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517843072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828054 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828029.845802:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828029.845804:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828029.845807:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517843072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828030.164781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787828030.677033:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828030.868825:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828030.868907:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828031.188787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787828031.700884:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828031.892746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828031.892746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828032.212772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787828032.724855:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828032.916752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828032.916814:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828033.236772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828033.748878:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828033.748932:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828033.940844:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828033.940902:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828034.260798:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828034.772752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828034.964813:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828034.964855:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828034.965087:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517843200:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828034.965182:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828034.965246:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828034] req@ffff8b2e8803b0c0 x1874672517843200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828034.965283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517843200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828034.965305:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828034.965314:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828034.965327:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517843200:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828034.965386:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517843328:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828034.965443:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828034.965468:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828034] req@ffff8b2e88038f00 x1874672517843328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828034.965515:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517843328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828034.965543:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828034.965552:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828034.965571:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517843328:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828034.965589:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517843456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828034.965631:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828034.965696:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828034] req@ffff8b2e8803b840 x1874672517843456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828034.965733:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517843456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828034.965808:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828034.965818:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828034.965827:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517843456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828034.965843:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517843584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828034.965886:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828034.965909:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828034] req@ffff8b2e8803a580 x1874672517843584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828034.965925:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517843584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828034.965936:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828034.965940:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828034.965945:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517843584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828034.965967:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517843712:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828034.965993:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828034.966006:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828034] req@ffff8b2e8803b0c0 x1874672517843712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828034.966021:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517843712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828034.966032:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828034.966046:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828034.966049:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517843712:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828034.966056:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517843840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828034.966074:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828034.966085:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828034] req@ffff8b2e880383c0 x1874672517843840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828034.966099:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517843840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828034.966108:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828034.966112:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828034.966115:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517843840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828034.966122:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517843968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828034.966139:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828034.966150:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828034/real 1787828034] req@ffff8b2e88039680 x1874672517843968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828034.966164:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517843968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828059 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828034.966174:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828034.966177:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828034.966180:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517843968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828035.284822:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828035.796799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828035.988748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828035.988815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828036.308821:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787828036.820935:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828036.821444:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828037.012750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828037.012750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828037.332833:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828037.844926:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787828038.036817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828038.036832:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828038.356811:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828038.868770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828039.060721:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828039.060721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828039.380801:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828039.892878:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:2.0:1787828040.084831:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828040.084847:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787828040.084852:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828040.084855:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828040.084894:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1046 00000100:00080000:2.0:1787828040.084919:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787828040.084922:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828040.084937:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828040.084986:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517844096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828040.085035:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828040.085039:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828040.085041:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828040.085046:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1046 00000100:00080000:2.0:1787828040.085050:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828040.085054:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828040.085084:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828040.085106:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517844096:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828040.085107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517844224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828040.085131:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828040.085135:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828040.085137:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828040.085143:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1046 00000100:00080000:2.0:1787828040.085147:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828040.085150:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828040.085163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828040.085178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517844352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828040.085189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828040.085193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828040.085196:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828040.085200:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1046 00000100:00080000:2.0:1787828040.085204:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828040.085207:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828040.085214:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828040.085222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828040.085235:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517844480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828040.085246:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828040.085250:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828040.085252:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828040.085256:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1046 00000100:00080000:2.0:1787828040.085261:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828040.085270:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828040.085271:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828040/real 1787828040] req@ffff8b2e8801c780 x1874672517844096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828040.085275:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828040.085296:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517844608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828040.085306:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517844096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828040.085313:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828040.085325:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828040.085330:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828040.085340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828040.085340:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1046 00000100:00080000:2.0:1787828040.085347:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828040.085353:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787828040.085359:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828040.085392:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517844096:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787828040.085401:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828040.085417:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517844736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828040.085422:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517844224:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828040.085441:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828040.085444:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828040.085446:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828040.085452:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1046 00000100:00080000:2.0:1787828040.085474:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828040.085481:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828040.085487:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828040.085496:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828040.085509:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828040/real 1787828040] req@ffff8b2e8801ed00 x1874672517844224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828040.085516:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517844864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828040.085551:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517844224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828040.085581:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828040.085586:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828040.085590:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517844224:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828040.085598:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517844352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828040.085616:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828040.085649:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828040/real 1787828040] req@ffff8b2e8801cb40 x1874672517844352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828040.085662:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517844352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828040.085671:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828040.085674:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828040.085677:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517844352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828040.085683:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517844480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828040.085700:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828040.085775:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828040/real 1787828040] req@ffff8b2e8801c3c0 x1874672517844480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828040.085810:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517844480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828040.085825:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828040.085829:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828040.085832:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517844480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828040.085840:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517844608:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828040.085866:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828040.085879:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828040/real 1787828040] req@ffff8b2e8801d680 x1874672517844608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828040.085892:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517844608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828040.085902:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828040.085905:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828040.085908:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517844608:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828040.085927:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517844736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828040.085951:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828040.085968:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828040/real 1787828040] req@ffff8b2e8801de00 x1874672517844736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828040.086003:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517844736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828040.086016:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828040.086021:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828040.086025:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517844736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828040.086034:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517844864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828040.086078:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828040.086090:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828040/real 1787828040] req@ffff8b2e8801f0c0 x1874672517844864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828040.086103:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517844864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828065 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828040.086112:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828040.086115:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828040.086118:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517844864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828040.404809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828040.916850:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828041.108860:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828041.108907:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828041.428799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828041.940835:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787828042.132836:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828042.132878:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828042.452843:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828042.964853:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787828043.156754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828043.156809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828043.988782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828044.180731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828044.180804:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828045.012812:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828045.012864:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828045.204756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828045.204756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828045.204888:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517844992:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828045.204953:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828045.204994:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828045/real 1787828045] req@ffff8b2e88039e00 x1874672517844992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828045.205022:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517844992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828045.205039:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828045.205046:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828045.205053:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517844992:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828045.205078:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517845120:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828045.205110:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828045.205122:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828045/real 1787828045] req@ffff8b2e8803a1c0 x1874672517845120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828045.205130:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517845120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828045.205136:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828045.205139:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828045.205141:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517845120:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828045.205145:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517845248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828045.205155:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828045.205161:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828045/real 1787828045] req@ffff8b2e8803b480 x1874672517845248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828045.205168:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517845248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828045.205184:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828045.205186:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828045.205187:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517845248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828045.205191:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517845376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828045.205201:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828045.205206:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828045/real 1787828045] req@ffff8b2e88039a40 x1874672517845376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828045.205213:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517845376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828045.205219:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828045.205221:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828045.205222:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517845376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828045.205226:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517845504:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828045.205235:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828045.205240:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828045/real 1787828045] req@ffff8b2e880392c0 x1874672517845504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828045.205247:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517845504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828045.205252:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828045.205254:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828045.205256:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517845504:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828045.205259:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517845632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828045.205268:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828045.205273:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828045/real 1787828045] req@ffff8b2e88039680 x1874672517845632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828045.205281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517845632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828045.205286:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828045.205287:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828045.205289:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517845632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828045.205299:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517845760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828045.205309:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828045.205315:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828045/real 1787828045] req@ffff8b2e880383c0 x1874672517845760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828045.205322:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517845760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828070 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828045.205327:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828045.205329:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828045.205331:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517845760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828046.036724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828046.228812:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828046.228842:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828047.060786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828047.252834:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828047.252834:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828048.084763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828048.084895:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828048.276791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828048.276827:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828049.108779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828049.300780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828049.300805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828050.132822:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828050.324816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828050.324828:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828050.325064:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517845888:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828050.325182:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828050.325257:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff8b2e8803b0c0 x1874672517845888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828050.325330:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517845888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828050.325353:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828050.325364:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828050.325375:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517845888:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828050.325426:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517846016:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828050.325478:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828050.325501:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff8b2e8803a580 x1874672517846016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828050.325516:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517846016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828050.325527:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828050.325531:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828050.325534:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517846016:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828050.325541:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517846144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828050.325561:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828050.325572:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff8b2e8803b840 x1874672517846144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828050.325586:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517846144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828050.325620:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828050.325624:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828050.325628:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517846144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828050.325635:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517846272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828050.325655:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828050.325737:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff8b2e88038f00 x1874672517846272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828050.325755:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517846272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828050.325766:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828050.325769:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828050.325773:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517846272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828050.325781:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517846400:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828050.325818:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828050.325831:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff8b2e880383c0 x1874672517846400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828050.325845:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517846400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828050.325855:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828050.325858:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828050.325862:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517846400:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828050.325882:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517846528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828050.325902:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828050.325915:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff8b2e88039680 x1874672517846528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828050.325929:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517846528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828050.325938:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828050.325941:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828050.325945:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517846528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828050.325966:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517846656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828050.325984:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828050.325994:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828050/real 1787828050] req@ffff8b2e880392c0 x1874672517846656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828050.326007:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517846656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828075 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828050.326018:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828050.326023:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828050.326028:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517846656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828051.156788:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828051.156914:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828051.348824:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828051.348843:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828052.180722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828052.372734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828052.372735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828053.204786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828053.396807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828053.396813:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828054.228795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828054.228944:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828054.420839:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828054.420848:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828055.252824:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828055.444766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828055.444786:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828055.444799:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828055.444806:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828055.444817:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1061 00000100:00080000:2.0:1787828055.444830:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828055.444839:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828055.444885:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517846784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828055.444915:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828055.444919:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828055.444921:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828055.444925:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1061 00000100:00080000:2.0:1787828055.444929:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828055.444933:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828055.444945:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828055.444962:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517846784:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828055.444963:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517846912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828055.444983:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828055.444987:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828055.444989:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828055.444993:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1061 00000100:00080000:2.0:1787828055.444997:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828055.445000:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828055.445006:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828055.445019:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517847040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828055.445028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828055.445031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828055.445033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828055.445036:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1061 00000100:00080000:2.0:1787828055.445040:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828055.445044:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828055.445049:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828055.445057:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828055.445070:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517847168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828055.445079:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828055.445083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828055.445085:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828055.445090:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1061 00000100:00080000:2.0:1787828055.445095:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828055.445099:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff8b2e8801f0c0 x1874672517846784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828055.445102:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828055.445112:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828055.445129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517847296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828055.445139:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828055.445146:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517846784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828055.445147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828055.445154:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828055.445167:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1061 00000100:00080000:2.0:1787828055.445179:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828055.445183:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828055.445190:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828055.445194:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828055.445204:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517846784:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787828055.445206:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828055.445230:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517847424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828055.445237:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517846912:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828055.445254:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828055.445258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828055.445261:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828055.445267:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1061 00000100:00080000:2.0:1787828055.445274:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828055.445283:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787828055.445286:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787828055.445292:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828055.445309:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff8b2e8801de00 x1874672517846912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828055.445311:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517847552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828055.445360:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517846912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828055.445378:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828055.445385:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828055.445391:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517846912:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828055.445399:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517847040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828055.445428:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828055.445441:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff8b2e8801d680 x1874672517847040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828055.445463:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517847040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828055.445472:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828055.445475:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828055.445479:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517847040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828055.445485:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517847168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828055.445500:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828055.445510:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff8b2e8801c3c0 x1874672517847168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828055.445521:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517847168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828055.445529:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828055.445532:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828055.445535:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517847168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828055.445541:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517847296:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828055.445556:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828055.445565:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff8b2e8801cb40 x1874672517847296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828055.445584:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517847296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828055.445593:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828055.445595:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828055.445598:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517847296:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828055.445616:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517847424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828055.445680:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828055.445696:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff8b2e8801ed00 x1874672517847424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828055.445709:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517847424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828055.445718:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828055.445721:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828055.445725:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517847424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828055.445731:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517847552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828055.445751:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828055.445760:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828055/real 1787828055] req@ffff8b2e8801c780 x1874672517847552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828055.445771:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517847552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828080 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828055.445786:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828055.445791:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828055.445797:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517847552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828056.276744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828056.468772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828057.300795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828057.300851:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828057.492747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828058.324874:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828058.516823:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828058.516833:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828059.348787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828059.540755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828059.540918:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828060.372864:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828060.372925:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828060.564831:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828060.564880:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828060.565228:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517847680:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828060.565444:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517847808:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828060.565520:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517847936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828060.565583:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517848064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828060.565750:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517848192:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828060.565816:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517848320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828060.565855:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517848448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828061.396813:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828061.588880:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828061.588881:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828062.420809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828062.612808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828062.612818:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828063.444783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828063.444876:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828063.636780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828063.636780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828063.637460:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828063] req@ffff8b2e88039e00 x1874672517848064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828063.637533:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517848064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828063.637561:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828063.637572:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828063.637622:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517848064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828063.637639:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828063] req@ffff8b2e880392c0 x1874672517848192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828063.637711:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517848192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828063.637729:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828063.637739:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828063.637749:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517848192:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828063.637767:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828063] req@ffff8b2e8803b480 x1874672517847808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828063.637800:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517847808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828063.637820:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828063.637828:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828063.637835:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517847808:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828063.637849:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828063] req@ffff8b2e8803a1c0 x1874672517847936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828063.637875:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517847936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828063.637891:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828063.637897:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828063.637904:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517847936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828063.637916:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828063] req@ffff8b2e88039a40 x1874672517847680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828063.637942:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517847680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828063.637958:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828063.637965:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828063.637972:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517847680:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828063.638030:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828063] req@ffff8b2e880383c0 x1874672517848448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828063.638057:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517848448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828063.638072:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828063.638080:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828063.638086:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517848448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828063.638098:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828060/real 1787828063] req@ffff8b2e88039680 x1874672517848320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828063.638123:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517848320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828085 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828063.638141:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828063.638148:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828063.638155:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517848320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828064.468833:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828064.660778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828064.660788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828065.492799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828065.684785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828065.684789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828065.685009:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517848576:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828065.685118:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828065.685186:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff8b2e88038f00 x1874672517848576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828065.685234:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517848576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828065.685271:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828065.685284:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828065.685303:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517848576:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828065.685347:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517848704:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828065.685409:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828065.685440:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff8b2e8803b840 x1874672517848704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828065.685487:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517848704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828065.685516:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828065.685528:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828065.685542:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517848704:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828065.685591:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517848832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828065.685679:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828065.685709:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff8b2e8803a580 x1874672517848832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828065.685730:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517848832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828065.685743:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828065.685748:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828065.685752:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517848832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828065.685762:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517848960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828065.685795:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828065.685814:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff8b2e8803b0c0 x1874672517848960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828065.685831:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517848960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828065.685844:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828065.685850:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828065.685855:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517848960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828065.685864:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517849088:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828065.685896:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828065.685914:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff8b2e88039680 x1874672517849088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828065.685932:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517849088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828065.685956:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828065.685961:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828065.685965:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517849088:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828065.685995:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517849216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828065.686029:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828065.686048:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff8b2e880383c0 x1874672517849216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828065.686068:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517849216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828065.686081:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828065.686086:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828065.686091:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517849216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828065.686100:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517849344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828065.686130:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828065.686148:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828065/real 1787828065] req@ffff8b2e88039a40 x1874672517849344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828065.686179:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517849344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828090 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828065.686192:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828065.686197:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828065.686211:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517849344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828066.516777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828066.516880:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828066.708804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828066.708804:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828067.540771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828067.732786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828067.732796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828068.564808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828068.756800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828068.756801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828069.588711:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828069.780761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828069.780788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828070.612819:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828070.804832:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828070.804832:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828070.805117:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517849472:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828070.805241:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828070.805351:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff8b2e8803a1c0 x1874672517849472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828070.805419:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517849472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828070.805502:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828070.805553:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828070.805576:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517849472:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828070.805704:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517849600:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828070.805770:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828070.805795:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff8b2e8803b480 x1874672517849600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828070.805814:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517849600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828070.805827:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828070.805832:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828070.805837:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517849600:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828070.805845:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517849728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828070.805871:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828070.805884:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff8b2e880392c0 x1874672517849728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828070.805899:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517849728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828070.805909:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828070.805913:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828070.805916:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517849728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828070.805924:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517849856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828070.805942:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828070.805953:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff8b2e88039e00 x1874672517849856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828070.805967:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517849856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828070.805986:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828070.805990:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828070.805994:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517849856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828070.806001:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517849984:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828070.806019:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828070.806030:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff8b2e88039a40 x1874672517849984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828070.806042:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517849984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828070.806053:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828070.806057:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828070.806061:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517849984:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828070.806068:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517850112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828070.806085:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828070.806096:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff8b2e880383c0 x1874672517850112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828070.806108:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517850112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828070.806119:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828070.806122:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828070.806127:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517850112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828070.806150:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517850240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828070.806170:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828070.806181:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828070/real 1787828070] req@ffff8b2e88039680 x1874672517850240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828070.806195:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517850240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828095 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828070.806205:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828070.806209:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828070.806212:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517850240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828071.636805:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828071.828753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828071.828817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828072.660802:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828072.852764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828072.852795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828073.684770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828073.876757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828073.876784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828074.708809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828074.900796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828074.900816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828075.732742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828075.924721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828075.924799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828075.925049:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517850368:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828075.925168:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828075.925249:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff8b2e8803b0c0 x1874672517850368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828075.925299:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517850368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828075.925318:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828075.925335:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828075.925349:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517850368:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828075.925395:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517850496:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828075.925436:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828075.925466:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff8b2e8803a580 x1874672517850496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828075.925527:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517850496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828075.925566:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828075.925575:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828075.925583:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517850496:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828075.925594:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517850624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828075.925621:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828075.925635:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff8b2e8803b840 x1874672517850624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828075.925680:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517850624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828075.925694:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828075.925700:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828075.925706:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517850624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828075.925716:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517850752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828075.925743:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828075.925760:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff8b2e88038f00 x1874672517850752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828075.925779:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517850752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828075.925790:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828075.925794:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828075.925798:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517850752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828075.925805:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517850880:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828075.925824:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828075.925835:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff8b2e88039680 x1874672517850880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828075.925849:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517850880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828075.925859:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828075.925863:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828075.925866:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517850880:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828075.925893:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517851008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828075.925932:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828075.925956:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff8b2e880383c0 x1874672517851008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828075.925982:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517851008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828075.926001:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828075.926007:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828075.926011:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517851008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828075.926019:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517851136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828075.926048:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828075.926063:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828075/real 1787828075] req@ffff8b2e88039a40 x1874672517851136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828075.926078:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517851136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828100 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828075.926088:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828075.926092:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828075.926096:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517851136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828076.756791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828076.948806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828076.948852:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828077.780831:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828077.972790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828077.972790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828078.804801:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828078.996789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828078.996852:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828079.828797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828080.020798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828080.020832:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828080.852811:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:1.0:1787828081.044851:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828081.044869:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828081.044872:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828081.044892:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828081.044902:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828081.044920:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1087 00000100:00080000:1.0:1787828081.044941:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828081.044957:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828081.045025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517851264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828081.045069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828081.045075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828081.045077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828081.045083:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1087 00000100:00080000:1.0:1787828081.045088:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828081.045093:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828081.045109:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828081.045124:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517851392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828081.045140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828081.045145:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828081.045148:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828081.045153:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1087 00000100:00080000:1.0:1787828081.045158:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828081.045163:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828081.045169:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828081.045182:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517851520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828081.045194:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517851264:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828081.045200:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828081.045212:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828081.045215:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828081.045222:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1087 00000100:00080000:1.0:1787828081.045226:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828081.045231:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828081.045237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828081.045254:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517851648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828081.045266:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828081.045271:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828081.045273:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828081.045278:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1087 00000100:00080000:1.0:1787828081.045305:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828081.045310:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828081.045316:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828081.045325:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828081.045338:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517851776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828081.045368:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828081.045374:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828081.045378:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828081.045388:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1087 00000100:00080000:1.0:1787828081.045397:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828081.045405:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828081/real 1787828081] req@ffff8b2e88077840 x1874672517851264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828081.045414:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828081.045437:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828081.045449:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517851264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828081.045497:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517851904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828081.045506:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828081.045524:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828081.045536:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517851264:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828081.045537:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828081.045550:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828081.045557:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828081.045567:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517851392:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828081.045575:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1087 00000100:00080000:1.0:1787828081.045589:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828081.045598:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828081.045612:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828081.045635:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828081.045643:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517852032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828081.045738:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828081/real 1787828081] req@ffff8b2e880752c0 x1874672517851392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828081.045768:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517851392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828081.045792:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828081.045798:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828081.045803:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517851392:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828081.045840:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517851520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828081.045874:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828081.045890:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828081/real 1787828081] req@ffff8b2e88075e00 x1874672517851520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828081.045906:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517851520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828081.045917:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828081.045921:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828081.045925:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517851520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828081.045934:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517851648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828081.045952:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828081.045964:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828081/real 1787828081] req@ffff8b2e88076d00 x1874672517851648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828081.045978:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517851648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828081.045988:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828081.045992:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828081.045995:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517851648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828081.046003:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517851776:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828081.046021:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828081.046032:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828081/real 1787828081] req@ffff8b2e88074000 x1874672517851776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828081.046046:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517851776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828081.046057:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828081.046060:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828081.046064:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517851776:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828081.046074:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517851904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828081.046109:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828081.046132:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828081/real 1787828081] req@ffff8b2e880761c0 x1874672517851904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828081.046158:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517851904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828081.046176:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828081.046183:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828081.046189:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517851904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828081.046198:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517852032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828081.046227:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828081.046241:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828081/real 1787828081] req@ffff8b2e880770c0 x1874672517852032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828081.046256:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517852032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828106 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828081.046267:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828081.046272:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828081.046275:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517852032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828081.878002:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828082.068779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828082.068796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828082.900789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828083.092737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828083.092791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828083.924776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828084.116782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828084.116840:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828084.948713:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828085.140712:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828085.140803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828085.972752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828086.164798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828086.164814:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828086.164839:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828086.164859:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828086.164870:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828086.164892:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1092 00000100:00080000:1.0:1787828086.164916:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828086.164929:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828086.165012:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517852160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828086.165058:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828086.165063:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828086.165067:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828086.165073:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1092 00000100:00080000:1.0:1787828086.165079:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828086.165085:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828086.165104:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828086.165132:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517852160:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828086.165134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517852288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828086.165173:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828086.165181:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828086.165187:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828086.165198:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1092 00000100:00080000:1.0:1787828086.165233:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828086.165246:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828086.165254:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828086.165267:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828086.165293:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517852416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828086.165309:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828086.165317:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828086/real 1787828086] req@ffff8b2e88074b40 x1874672517852160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828086.165320:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828086.165326:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828086.165345:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1092 00000100:00080000:1.0:1787828086.165362:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828086.165376:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828086.165383:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828086.165388:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517852160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828086.165408:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517852544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828086.165415:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828086.165457:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828086.165460:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828086.165478:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787828086.165492:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517852160:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828086.165494:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828086.165508:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1092 00000100:00080000:1.0:1787828086.165525:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828086.165535:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828086.165544:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828086.165562:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517852288:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828086.165568:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517852672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828086.165590:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828086.165595:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828086.165598:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828086.165604:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1092 00000100:00080000:1.0:1787828086.165611:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828086.165620:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828086.165628:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828086.165640:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828086.165701:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517852800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828086.165704:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828086/real 1787828086] req@ffff8b2e88076580 x1874672517852288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828086.165780:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828086.165798:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828086.165801:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828086.165810:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1092 00000100:00080000:1.0:1787828086.165816:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828086.165821:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517852288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828086.165832:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828086.165860:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828086.165865:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828086.165874:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828086.165883:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517852288:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828086.165899:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517852416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828086.165899:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517852928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828086.165959:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828086.165988:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828086/real 1787828086] req@ffff8b2e880770c0 x1874672517852416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828086.166005:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517852416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828086.166024:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828086.166029:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828086.166033:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517852416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828086.166041:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517852544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828086.166061:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828086.166073:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828086/real 1787828086] req@ffff8b2e880761c0 x1874672517852544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828086.166087:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517852544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828086.166098:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828086.166101:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828086.166105:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517852544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828086.166129:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517852672:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828086.166149:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828086.166162:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828086/real 1787828086] req@ffff8b2e88074000 x1874672517852672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828086.166176:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517852672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828086.166187:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828086.166190:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828086.166195:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517852672:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828086.166203:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517852800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828086.166221:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828086.166232:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828086/real 1787828086] req@ffff8b2e88076d00 x1874672517852800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828086.166246:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517852800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828086.166256:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828086.166259:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828086.166263:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517852800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828086.166270:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517852928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828086.166288:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828086.166299:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828086/real 1787828086] req@ffff8b2e88075e00 x1874672517852928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828086.166313:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517852928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828111 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828086.166322:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828086.166326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828086.166329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517852928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828086.996772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828087.188832:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828087.188864:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828088.020789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828088.212874:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828089.044806:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828089.236940:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828089.236940:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828090.068795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787828090.260817:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828090.260827:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828091.092791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828091.284817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828091.284859:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.284879:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828091.284889:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.284910:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1097 00000100:00080000:1.0:1787828091.284932:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.284945:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828091.285007:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517853056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828091.285046:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.285051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828091.285055:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.285060:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1097 00000100:00080000:1.0:1787828091.285065:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.285073:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828091.285088:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828091.285106:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517853184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828091.285117:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517853056:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828091.285130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.285136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828091.285138:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.285142:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1097 00000100:00080000:1.0:1787828091.285146:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.285151:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828091.285156:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828091.285170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517853312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828091.285179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.285184:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828091.285186:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.285191:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1097 00000100:00080000:1.0:1787828091.285207:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.285212:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828091.285219:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828091.285221:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828091.285244:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517853440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828091.285260:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.285265:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828091.285267:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.285275:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1097 00000100:00080000:1.0:1787828091.285280:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.285288:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828091.285295:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828091.285299:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff8b2e880752c0 x1874672517853056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828091.285316:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517853568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828091.285347:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828091.285352:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828091.285354:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.285361:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1097 00000100:00080000:1.0:1787828091.285370:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828091.285373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517853056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828091.285379:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828091.285407:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828091.285439:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828091.285439:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517853696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828091.285463:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828091.285486:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828091.285488:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517853056:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828091.285511:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828091.285515:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828091.285526:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1097 00000100:00080000:1.0:1787828091.285532:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828091.285537:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828091.285545:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828091.285555:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517853184:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828091.285566:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517853824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828091.285603:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828091.285628:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff8b2e88077840 x1874672517853184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828091.285680:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517853184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828091.285693:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828091.285698:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828091.285703:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517853184:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828091.285712:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517853312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828091.285748:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828091.285771:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff8b2e88074780 x1874672517853312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828091.285796:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517853312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828091.285813:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828091.285821:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828091.285827:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517853312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828091.285841:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517853440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828091.285875:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828091.285891:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff8b2e88076940 x1874672517853440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828091.285905:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517853440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828091.285916:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828091.285920:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828091.285924:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517853440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828091.285932:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517853568:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828091.285952:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828091.285963:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff8b2e88074f00 x1874672517853568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828091.285986:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517853568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828091.285997:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828091.286000:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828091.286004:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517853568:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828091.286012:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517853696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828091.286033:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828091.286044:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff8b2e88075680 x1874672517853696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828091.286058:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517853696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828091.286068:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828091.286072:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828091.286075:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517853696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828091.286101:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517853824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828091.286121:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828091.286132:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828091/real 1787828091] req@ffff8b2e88075a40 x1874672517853824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828091.286146:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517853824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828116 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828091.286165:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828091.286169:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828091.286172:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517853824:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787828092.116972:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828092.308724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828093.140785:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828093.332740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828094.164821:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828094.356732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828095.188991:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828095.380844:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828096.212903:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828096.404800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828096.404850:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828096.404858:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828096.404865:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828096.404880:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1102 00000100:00080000:0.0:1787828096.404897:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828096.404908:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828096.404970:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517853952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828096.405015:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828096.405020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828096.405023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828096.405042:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1102 00000100:00080000:0.0:1787828096.405048:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828096.405055:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828096.405070:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828096.405098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517854080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828096.405116:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828096.405121:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828096.405124:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828096.405129:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1102 00000100:00080000:0.0:1787828096.405133:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828096.405138:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828096.405144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828096.405160:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517854208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828096.405173:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828096.405178:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828096.405180:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828096.405184:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1102 00000100:00080000:0.0:1787828096.405199:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828096.405204:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828096.405209:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828096.405222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517854336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828096.405245:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828096.405251:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828096.405254:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828096.405262:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1102 00000100:00080000:0.0:1787828096.405268:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828096.405274:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828096.405280:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828096.405295:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517854464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828096.405308:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828096.405312:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828096.405315:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828096.405324:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1102 00000100:00080000:0.0:1787828096.405333:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828096.405372:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828096.405381:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828096.405403:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517854592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828096.405434:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828096.405445:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828096.405450:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828096.405460:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1102 00000100:00080000:0.0:1787828096.405474:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828096.405479:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828096.405485:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828096.405498:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517854720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828097.236847:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828097.428762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828098.260907:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828098.452802:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828099.284849:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828099.476798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828099.476805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828100.308916:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828100.500803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828100.500810:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828101.332836:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828101.524724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828101.524783:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828101.524790:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828101.524794:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828101.524804:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1107 00000100:00080000:0.0:1787828101.524816:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828101.524828:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828101.524866:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517854848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828101.524889:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828101.524892:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828101.524894:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828101.524898:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1107 00000100:00080000:0.0:1787828101.524901:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828101.524904:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828101.524914:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828101.524930:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517854976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828101.524946:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828101.524948:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828101.524951:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828101.524954:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1107 00000100:00080000:0.0:1787828101.524957:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828101.524960:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828101.524964:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828101.524974:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517855104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828101.524982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828101.524985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828101.524987:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828101.524990:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1107 00000100:00080000:0.0:1787828101.524992:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828101.524995:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828101.525000:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828101.525009:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517855232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828101.525026:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828101.525030:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828101.525032:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828101.525037:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1107 00000100:00080000:0.0:1787828101.525041:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828101.525044:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828101.525048:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828101.525058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517855360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828101.525065:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828101.525068:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828101.525069:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828101.525072:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1107 00000100:00080000:0.0:1787828101.525076:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828101.525083:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828101.525092:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828101.525109:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517855488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828101.525119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828101.525129:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828101.525133:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828101.525143:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1107 00000100:00080000:0.0:1787828101.525150:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828101.525156:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828101.525163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828101.525189:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517855616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828102.356922:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828102.548826:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828103.380909:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828103.572799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828104.404877:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828104.596808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828105.428885:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828105.620792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828106.452968:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828106.644844:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828106.644848:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828106.644863:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828106.644870:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828106.644906:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1113 00000100:00080000:0.0:1787828106.644925:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828106.644937:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828106.645000:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517855744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828106.645046:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828106.645052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828106.645055:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828106.645061:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1113 00000100:00080000:0.0:1787828106.645066:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828106.645072:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828106.645088:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828106.645114:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517855872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828106.645141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828106.645146:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828106.645148:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828106.645153:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1113 00000100:00080000:0.0:1787828106.645159:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828106.645163:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828106.645170:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828106.645189:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517856000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828106.645210:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828106.645215:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828106.645219:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828106.645224:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1113 00000100:00080000:0.0:1787828106.645228:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828106.645232:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828106.645239:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828106.645262:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517856128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828106.645302:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828106.645308:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828106.645311:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828106.645318:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1113 00000100:00080000:0.0:1787828106.645324:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828106.645332:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828106.645339:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828106.645365:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517856256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828106.645408:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828106.645413:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828106.645420:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828106.645437:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1113 00000100:00080000:0.0:1787828106.645448:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828106.645458:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828106.645474:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828106.645498:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517856384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828106.645512:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828106.645517:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828106.645520:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828106.645526:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1113 00000100:00080000:0.0:1787828106.645531:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828106.645542:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828106.645549:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828106.645563:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517856512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828107.477009:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828107.668858:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828108.500832:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828108.692780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828109.524819:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828109.716770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828109.716842:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828110.548856:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828110.740834:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828110.740849:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828111.572865:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828111.764824:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828111.764848:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828111.765064:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517856640:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828111.765161:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828111.765260:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff8b2e88039e00 x1874672517856640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828111.765310:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517856640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828111.765334:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828111.765344:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828111.765353:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517856640:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828111.765394:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517856768:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828111.765484:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828111.765523:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff8b2e880392c0 x1874672517856768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828111.765568:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517856768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828111.765595:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828111.765607:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828111.765620:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517856768:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828111.765641:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517856896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828111.765714:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828111.765733:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff8b2e8803b480 x1874672517856896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828111.765749:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517856896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828111.765769:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828111.765781:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828111.765791:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517856896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828111.765806:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517857024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828111.765848:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828111.765884:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff8b2e8803a1c0 x1874672517857024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828111.765903:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517857024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828111.765914:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828111.765918:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828111.765923:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517857024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828111.765945:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517857152:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828111.765973:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828111.765987:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff8b2e88039a40 x1874672517857152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828111.766002:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517857152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828111.766014:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828111.766018:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828111.766022:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517857152:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828111.766028:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517857280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828111.766046:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828111.766057:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff8b2e880383c0 x1874672517857280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828111.766071:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517857280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828111.766089:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828111.766093:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828111.766097:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517857280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828111.766103:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517857408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828111.766120:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828111.766130:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828111/real 1787828111] req@ffff8b2e88039680 x1874672517857408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828111.766144:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517857408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828136 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828111.766154:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828111.766158:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828111.766161:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517857408:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787828112.596944:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828112.788795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828112.788817:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828113.620878:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828113.812792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828113.812797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828114.644878:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828114.836758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828114.836769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828115.668882:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828115.860707:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828115.860726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828116.692935:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828116.884816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828116.884827:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828116.885125:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517857536:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828116.885295:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828116.885382:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff8b2e88038f00 x1874672517857536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828116.885444:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517857536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828116.885467:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828116.885473:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828116.885487:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517857536:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828116.885541:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517857664:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828116.885587:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828116.885611:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff8b2e8803b840 x1874672517857664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828116.885633:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517857664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828116.885704:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828116.885720:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828116.885734:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517857664:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828116.885748:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517857792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828116.885790:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828116.885812:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff8b2e8803a580 x1874672517857792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828116.885829:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517857792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828116.885842:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828116.885847:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828116.885851:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517857792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828116.885859:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517857920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828116.885878:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828116.885889:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff8b2e8803b0c0 x1874672517857920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828116.885904:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517857920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828116.885914:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828116.885917:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828116.885921:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517857920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828116.885927:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517858048:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828116.885945:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828116.885956:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff8b2e88039680 x1874672517858048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828116.885971:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517858048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828116.885981:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828116.885984:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828116.885987:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517858048:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828116.886008:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517858176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828116.886027:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828116.886038:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff8b2e880383c0 x1874672517858176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828116.886052:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517858176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828116.886071:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828116.886075:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828116.886079:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517858176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828116.886085:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517858304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828116.886102:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828116.886113:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828116/real 1787828116] req@ffff8b2e88039a40 x1874672517858304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828116.886126:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517858304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828141 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828116.886136:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828116.886140:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828116.886143:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517858304:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787828117.716858:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828117.908790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828117.908828:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828118.740898:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828118.932849:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828118.932865:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828119.764926:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828119.956770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828119.956819:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828120.788897:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828120.980775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828120.980823:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828121.812904:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828122.004779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828122.004800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828122.005078:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517858432:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828122.005205:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828122.005290:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828122/real 1787828122] req@ffff8b2e8803a1c0 x1874672517858432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828122.005326:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517858432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828122.005345:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828122.005353:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828122.005361:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517858432:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828122.005396:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517858560:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828122.005439:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828122.005461:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828122/real 1787828122] req@ffff8b2e8803b480 x1874672517858560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828122.005486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517858560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828122.005504:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828122.005512:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828122.005519:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517858560:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828122.005531:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517858688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828122.005579:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828122.005604:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828122/real 1787828122] req@ffff8b2e880392c0 x1874672517858688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828122.005623:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517858688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828122.005633:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828122.005638:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828122.005642:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517858688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828122.005681:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517858816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828122.005710:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828122.005724:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828122/real 1787828122] req@ffff8b2e88039e00 x1874672517858816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828122.005740:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517858816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828122.005751:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828122.005754:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828122.005758:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517858816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828122.005765:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517858944:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828122.005784:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828122.005795:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828122/real 1787828122] req@ffff8b2e88039a40 x1874672517858944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828122.005810:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517858944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828122.005830:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828122.005833:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828122.005837:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517858944:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828122.005858:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517859072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828122.005877:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828122.005888:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828122/real 1787828122] req@ffff8b2e880383c0 x1874672517859072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828122.005901:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517859072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828122.005929:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828122.005933:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828122.005937:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517859072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828122.005943:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517859200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828122.005959:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828122.005970:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828122/real 1787828122] req@ffff8b2e88039680 x1874672517859200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828122.005983:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517859200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828147 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828122.005993:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828122.005997:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828122.006000:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517859200:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787828122.836932:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828123.028784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828123.028828:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828123.860976:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828124.052790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828124.052799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828124.884895:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828125.076755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828125.076759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828125.908885:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828126.100785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828126.100819:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828126.932917:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828127.124822:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828127.124823:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828127.125058:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517859328:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828127.125273:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517859456:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828127.125340:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517859584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828127.125392:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517859712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828127.125459:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517859840:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828127.125506:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517859968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828127.125554:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517860096:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787828127.956950:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828128.148858:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828128.148984:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828128.981001:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828129.172780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828129.172968:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828130.004928:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828130.196807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828130.196860:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:0.0:1787828130.197151:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:0.0:1787828130.197215:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828130.197236:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828130.197286:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828130.197296:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828130.197308:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828130.197322:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828130.197339:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828130.197361:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00010000:00010000:1.0:1787828131.028922:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828131.220782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828131.220798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828132.052963:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828132.244770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828132.244813:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828132.244822:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828132.244832:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828132.244845:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1138 00000100:00080000:2.0:1787828132.244859:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828132.244870:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828132.244919:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517860224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828132.244953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828132.244959:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828132.244961:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828132.244965:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1138 00000100:00080000:2.0:1787828132.244969:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828132.244973:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828132.244985:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828132.245000:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517860352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828132.245013:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828132.245032:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828132.245035:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828132.245040:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1138 00000100:00080000:2.0:1787828132.245044:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828132.245048:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828132.245054:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828132.245070:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517860480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828132.245080:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828132.245085:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828132.245088:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828132.245091:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1138 00000100:00080000:2.0:1787828132.245095:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828132.245127:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828132.245137:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828132.245151:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517860608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828132.245162:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828132.245166:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828132.245168:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828132.245173:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1138 00000100:00080000:2.0:1787828132.245176:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828132.245181:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828132.245186:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828132.245202:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517860736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828132.245246:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828132.245254:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828132.245257:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828132.245268:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1138 00000100:00080000:2.0:1787828132.245274:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828132.245283:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828132.245296:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828132.245324:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517860864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828132.245358:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828132.245363:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828132.245368:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828132.245375:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1138 00000100:00080000:2.0:1787828132.245382:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828132.245393:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828132.245403:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828132.245431:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517860992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828133.076970:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828133.268783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828134.100958:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828134.292827:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828135.124890:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828135.316785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828136.148852:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828136.340752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828137.172914:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828137.364818:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828137.364901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828137.364919:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828137.364928:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828137.364973:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1143 00000100:00080000:2.0:1787828137.364996:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828137.365010:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828137.365112:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517861120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828137.365167:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828137.365173:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828137.365176:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828137.365183:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1143 00000100:00080000:2.0:1787828137.365191:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828137.365198:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828137.365215:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828137.365241:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517861248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828137.365271:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828137.365276:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828137.365279:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828137.365285:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1143 00000100:00080000:2.0:1787828137.365290:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828137.365321:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828137.365329:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828137.365348:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517861376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828137.365364:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828137.365369:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828137.365371:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828137.365377:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1143 00000100:00080000:2.0:1787828137.365400:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828137.365407:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828137.365414:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828137.365436:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517861504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828137.365549:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828137.365580:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828137.365586:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828137.365608:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1143 00000100:00080000:2.0:1787828137.365625:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828137.365638:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828137.365647:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828137.365766:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517861632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828137.365839:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828137.365853:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828137.365883:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828137.365904:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1143 00000100:00080000:2.0:1787828137.365919:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828137.365934:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828137.365970:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828137.366017:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517861760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828137.366107:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828137.366119:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828137.366124:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828137.366137:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1143 00000100:00080000:2.0:1787828137.366158:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828137.366171:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828137.366183:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828137.366227:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517861888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828138.196985:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828138.388779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828139.220916:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828139.412791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:0.0:1787828140.244977:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828140.436815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828140.436903:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828141.268777:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828141.460734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828141.460744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828142.292953:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828142.484815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828142.484857:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828142.484872:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828142.484887:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828142.484897:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828142.484931:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1148 00000100:00080000:1.0:1787828142.484952:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828142.484969:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828142.485060:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517862016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828142.485123:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828142.485129:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828142.485132:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828142.485149:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1148 00000100:00080000:1.0:1787828142.485156:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828142.485164:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828142.485182:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828142.485206:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517862144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828142.485239:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828142.485254:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828142.485256:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828142.485264:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1148 00000100:00080000:1.0:1787828142.485270:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828142.485275:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828142.485281:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828142.485297:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517862272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828142.485309:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828142.485313:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828142.485316:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828142.485322:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1148 00000100:00080000:1.0:1787828142.485329:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828142.485361:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828142.485374:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828142.485405:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517862400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828142.485500:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828142.485516:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828142.485530:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828142.485544:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1148 00000100:00080000:1.0:1787828142.485551:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828142.485557:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828142.485568:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828142.485605:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517862528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828142.485643:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828142.485698:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828142.485702:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828142.485712:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1148 00000100:00080000:1.0:1787828142.485722:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828142.485732:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828142.485743:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828142.485795:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517862656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828142.485964:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828142.485971:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828142.485975:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828142.485986:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1148 00000100:00080000:1.0:1787828142.485996:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828142.486004:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828142.486054:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828142.486092:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517862784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787828143.316864:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828143.508749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828143.508784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828144.340761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828144.340799:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828144.532757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828144.532767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828145.364718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828145.364790:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828145.556705:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828145.556707:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828146.388756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828146.388781:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828146.580761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828146.580764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828147.412814:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828147.412837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828147.604751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828147.604788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828147.605141:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517862912:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828147.605249:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828147.605315:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff8b2e88039a40 x1874672517862912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828147.605361:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517862912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828147.605386:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828147.605400:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828147.605414:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517862912:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828147.605456:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517863040:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828147.605500:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828147.605519:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff8b2e8803b0c0 x1874672517863040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828147.605546:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517863040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828147.605608:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828147.605622:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828147.605628:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517863040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828147.605645:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517863168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828147.605734:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828147.605758:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff8b2e88038f00 x1874672517863168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828147.605791:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517863168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828147.605804:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828147.605810:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828147.605814:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517863168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828147.605823:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517863296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828147.605844:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828147.605856:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff8b2e8803b840 x1874672517863296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828147.605870:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517863296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828147.605880:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828147.605883:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828147.605886:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517863296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828147.605894:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517863424:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828147.605911:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828147.605922:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff8b2e8803a580 x1874672517863424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828147.605936:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517863424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828147.605947:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828147.605950:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828147.605954:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517863424:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828147.605975:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517863552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828147.606026:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828147.606043:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff8b2e880383c0 x1874672517863552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828147.606068:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517863552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828147.606084:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828147.606091:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828147.606097:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517863552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828147.606110:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517863680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828147.606150:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828147.606171:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828147/real 1787828147] req@ffff8b2e88039680 x1874672517863680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828147.606187:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517863680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828172 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828147.606206:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828147.606210:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828147.606214:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517863680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828148.436776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828148.436871:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828148.628785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828148.628822:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828149.460797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828149.460840:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828149.652756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828149.652780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:0.0:1787828150.484865:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828150.484968:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828150.676809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828151.508888:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828151.508903:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828151.700809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828152.532813:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828152.532874:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828152.724785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828152.724857:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828152.725140:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517863808:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828152.725272:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828152.725347:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff8b2e88039e00 x1874672517863808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828152.725400:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517863808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828152.725422:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828152.725436:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828152.725449:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517863808:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828152.725505:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517863936:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828152.725548:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828152.725571:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff8b2e880392c0 x1874672517863936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828152.725614:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517863936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828152.725681:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828152.725690:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828152.725698:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517863936:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828152.725719:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517864064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828152.725794:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828152.725835:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff8b2e8803b480 x1874672517864064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828152.725855:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517864064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828152.725866:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828152.725871:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828152.725875:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517864064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828152.725883:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517864192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828152.725908:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828152.725921:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff8b2e8803a1c0 x1874672517864192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828152.725936:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517864192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828152.725982:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828152.725992:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828152.725997:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517864192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828152.726006:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517864320:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828152.726034:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828152.726060:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff8b2e88039680 x1874672517864320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828152.726075:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517864320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828152.726087:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828152.726091:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828152.726095:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517864320:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828152.726117:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517864448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828152.726139:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828152.726151:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff8b2e880383c0 x1874672517864448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828152.726166:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517864448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828152.726176:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828152.726189:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828152.726194:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517864448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828152.726201:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517864576:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828152.726219:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828152.726230:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828152/real 1787828152] req@ffff8b2e8803a580 x1874672517864576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828152.726244:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517864576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828177 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828152.726254:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828152.726257:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828152.726261:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517864576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828153.556881:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828153.556892:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828153.748755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828153.748814:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828154.580829:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828154.580936:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828154.772782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828154.772845:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828155.604849:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828155.604880:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828155.796823:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828155.796832:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828156.628820:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828156.628878:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828156.820727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828156.820733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828157.652721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828157.652943:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828157.844787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828157.844840:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828157.845048:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517864704:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828157.845113:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828157.845164:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff8b2e8803b840 x1874672517864704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828157.845204:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517864704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828157.845241:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828157.845254:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828157.845266:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517864704:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828157.845286:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517864832:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828157.845314:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828157.845326:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff8b2e88038f00 x1874672517864832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828157.845336:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517864832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828157.845344:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828157.845347:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828157.845349:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517864832:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828157.845370:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517864960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828157.845385:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828157.845393:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff8b2e8803b0c0 x1874672517864960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828157.845403:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517864960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828157.845409:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828157.845412:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828157.845414:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517864960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828157.845419:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517865088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828157.845431:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828157.845437:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff8b2e88039a40 x1874672517865088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828157.845446:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517865088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828157.845453:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828157.845455:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828157.845457:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517865088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828157.845461:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517865216:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828157.845473:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828157.845480:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff8b2e8803a580 x1874672517865216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828157.845489:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517865216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828157.845495:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828157.845497:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828157.845500:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517865216:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828157.845504:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517865344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828157.845515:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828157.845522:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff8b2e880383c0 x1874672517865344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828157.845532:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517865344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828157.845538:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828157.845540:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828157.845542:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517865344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828157.845546:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517865472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828157.845569:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828157.845577:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828157/real 1787828157] req@ffff8b2e88039680 x1874672517865472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828157.845587:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517865472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828182 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828157.845593:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828157.845595:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828157.845598:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517865472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828158.676839:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828158.676844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828158.868763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828158.868786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828159.700731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828159.700768:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828159.892792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828159.892795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828160.724793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828160.724975:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828160.916780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828161.748776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828161.748843:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828161.940852:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828162.772866:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828162.772917:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828162.964860:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828162.964944:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828162.965180:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517865600:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828162.965274:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828162.965348:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff8b2e8803a1c0 x1874672517865600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828162.965385:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517865600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828162.965406:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828162.965415:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828162.965443:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517865600:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828162.965491:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517865728:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828162.965521:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828162.965539:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff8b2e8803b480 x1874672517865728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828162.965562:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517865728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828162.965576:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828162.965598:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828162.965614:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517865728:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828162.965622:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517865856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828162.965651:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828162.965662:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff8b2e880392c0 x1874672517865856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828162.965714:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517865856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828162.965722:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828162.965724:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828162.965727:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517865856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828162.965733:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517865984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828162.965750:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828162.965780:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff8b2e88039e00 x1874672517865984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828162.965793:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517865984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828162.965803:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828162.965807:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828162.965811:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517865984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828162.965817:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517866112:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828162.965855:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828162.965867:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff8b2e88039680 x1874672517866112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828162.965881:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517866112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828162.965941:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828162.965944:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828162.965947:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517866112:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828162.965952:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517866240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828162.965989:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828162.966006:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff8b2e880383c0 x1874672517866240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828162.966032:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517866240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828162.966043:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828162.966047:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828162.966051:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517866240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828162.966057:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517866368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828162.966079:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828162.966092:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828162/real 1787828162] req@ffff8b2e8803a580 x1874672517866368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828162.966105:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517866368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828187 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828162.966115:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828162.966118:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828162.966123:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517866368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828163.796825:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828163.796961:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828163.988777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828163.988816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828164.820833:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828164.820891:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828165.012781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828165.012853:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828165.844815:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828165.844908:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828166.036765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828166.036765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828166.868796:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828166.868940:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828167.060784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828167.060784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828167.892795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828167.892810:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828168.084772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828168.084789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828168.085047:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517866496:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828168.085180:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828168.085266:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828168/real 1787828168] req@ffff8b2e88039a40 x1874672517866496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828168.085329:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517866496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828168.085378:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828168.085388:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828168.085399:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517866496:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828168.085454:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517866624:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828168.085500:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828168.085515:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828168/real 1787828168] req@ffff8b2e8803b0c0 x1874672517866624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828168.085538:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517866624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828168.085555:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828168.085558:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828168.085561:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517866624:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828168.085568:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517866752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828168.085587:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828168.085597:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828168/real 1787828168] req@ffff8b2e88038f00 x1874672517866752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828168.085624:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517866752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828168.085632:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828168.085635:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828168.085638:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517866752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828168.085643:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517866880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828168.085658:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828168.085692:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828168/real 1787828168] req@ffff8b2e8803b840 x1874672517866880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828168.085704:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517866880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828168.085721:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828168.085723:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828168.085727:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517866880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828168.085733:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517867008:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828168.085749:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828168.085758:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828168/real 1787828168] req@ffff8b2e8803a580 x1874672517867008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828168.085786:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517867008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828168.085794:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828168.085797:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828168.085800:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517867008:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828168.085805:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517867136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828168.085820:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828168.085829:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828168/real 1787828168] req@ffff8b2e880383c0 x1874672517867136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828168.085873:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517867136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828168.085892:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828168.085895:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828168.085898:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517867136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828168.085904:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517867264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828168.085925:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828168.085957:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828168/real 1787828168] req@ffff8b2e88039680 x1874672517867264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828168.085972:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517867264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828193 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828168.085982:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828168.085985:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828168.085989:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517867264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828168.916745:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828168.916814:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828169.108799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828169.108799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828169.940737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828169.940810:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828170.132762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828170.132784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828170.964749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828170.964878:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828171.156783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828171.988721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828171.988908:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828172.180772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828173.012777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828173.012896:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828173.204792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828173.204840:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828173.204996:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517867392:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828173.205092:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828173.205146:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828173/real 1787828173] req@ffff8b2e88039e00 x1874672517867392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828173.205197:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517867392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828173.205218:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828173.205223:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828173.205228:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517867392:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828173.205261:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517867520:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828173.205304:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828173.205323:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828173/real 1787828173] req@ffff8b2e880392c0 x1874672517867520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828173.205359:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517867520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828173.205386:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828173.205401:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828173.205411:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517867520:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828173.205422:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517867648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828173.205467:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828173.205490:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828173/real 1787828173] req@ffff8b2e8803b480 x1874672517867648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828173.205517:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517867648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828173.205541:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828173.205549:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828173.205554:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517867648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828173.205561:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517867776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828173.205580:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828173.205589:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828173/real 1787828173] req@ffff8b2e8803a1c0 x1874672517867776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828173.205606:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517867776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828173.205613:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828173.205616:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828173.205619:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517867776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828173.205634:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517867904:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828173.205688:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828173.205700:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828173/real 1787828173] req@ffff8b2e88039e00 x1874672517867904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828173.205712:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517867904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828173.205721:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828173.205724:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828173.205727:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517867904:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828173.205733:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517868032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828173.205750:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828173.205758:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828173/real 1787828173] req@ffff8b2e88039680 x1874672517868032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828173.205769:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517868032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828173.205782:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828173.205785:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828173.205788:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517868032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828173.205793:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517868160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828173.205806:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828173.205838:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828173/real 1787828173] req@ffff8b2e880383c0 x1874672517868160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828173.205850:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517868160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828198 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828173.205859:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828173.205880:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828173.205884:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517868160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828174.036729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828174.036765:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828174.228716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828174.228723:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828175.060775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828175.060855:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828175.252793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828175.252831:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828176.084789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828176.084943:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828176.276760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828176.276855:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828177.108828:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828177.108954:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828177.300793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828177.300885:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828178.132803:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828178.132938:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828178.324881:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828178.324881:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828178.325107:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517868288:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828178.325214:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828178.325289:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff8b2e8803a580 x1874672517868288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828178.325332:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517868288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828178.325361:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828178.325372:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828178.325393:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517868288:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828178.325431:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517868416:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828178.325470:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828178.325493:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff8b2e8803b840 x1874672517868416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828178.325520:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517868416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828178.325533:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828178.325537:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828178.325568:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517868416:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828178.325578:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517868544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828178.325606:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828178.325626:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff8b2e88038f00 x1874672517868544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828178.325640:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517868544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828178.325650:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828178.325654:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828178.325658:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517868544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828178.325664:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517868672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828178.325721:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828178.325735:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff8b2e8803b0c0 x1874672517868672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828178.325750:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517868672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828178.325761:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828178.325765:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828178.325769:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517868672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828178.325797:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517868800:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828178.325843:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828178.325857:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff8b2e88039a40 x1874672517868800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828178.325880:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517868800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828178.325890:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828178.325893:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828178.325896:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517868800:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828178.325914:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517868928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828178.325932:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828178.325950:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff8b2e8803a580 x1874672517868928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828178.325971:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517868928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828178.325981:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828178.325983:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828178.325987:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517868928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828178.325992:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517869056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828178.326007:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828178.326015:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828178/real 1787828178] req@ffff8b2e880383c0 x1874672517869056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828178.326045:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517869056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828203 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828178.326055:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828178.326059:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828178.326062:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517869056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828179.156864:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828179.156971:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828179.348788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828179.348874:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828180.180758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828180.180848:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828180.372802:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828180.372802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828181.204737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828181.204848:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828181.396732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828182.228783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828182.228958:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828182.420763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828183.252846:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828183.252865:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828183.444843:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828183.444953:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828183.444963:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828183.444970:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828183.444983:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1189 00000100:00080000:2.0:1787828183.444997:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828183.445007:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828183.445053:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517869184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828183.445092:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828183.445097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828183.445100:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828183.445105:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1189 00000100:00080000:2.0:1787828183.445109:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828183.445114:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828183.445127:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828183.445143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517869312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828183.445163:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517869184:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787828183.445163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828183.445179:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828183.445184:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828183.445190:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1189 00000100:00080000:2.0:1787828183.445194:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828183.445198:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828183.445204:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828183.445222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517869440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828183.445234:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828183.445239:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828183.445242:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828183.445246:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1189 00000100:00080000:2.0:1787828183.445250:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828183.445254:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828183.445259:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828183.445270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517869568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828183.445281:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828183.445285:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828183.445287:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828183.445291:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1189 00000100:00080000:2.0:1787828183.445295:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828183.445299:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828183.445303:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828183.445313:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517869696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828183.445325:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828183.445328:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828183.445330:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828183.445337:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1189 00000800:00000100:0.0:1787828183.445345:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828183.445347:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828183.445358:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828183.445365:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828183.445380:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517869824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828183.445391:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828183.445396:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828183.445398:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828183.445403:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1189 00000100:00080000:2.0:1787828183.445409:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828183.445414:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff8b2e8801c780 x1874672517869184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828183.445420:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828183.445433:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828183.445458:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517869952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828183.445463:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517869184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828183.445508:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828183.445519:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828183.445528:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517869184:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828183.445560:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517869312:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828183.445592:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828183.445608:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff8b2e8801ed00 x1874672517869312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828183.445624:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517869312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828183.445636:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828183.445640:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828183.445644:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517869312:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828183.445688:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517869440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828183.445713:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828183.445727:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff8b2e8801cb40 x1874672517869440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828183.445774:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517869440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828183.445797:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828183.445801:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828183.445806:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517869440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828183.445815:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517869568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828183.445843:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828183.445858:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff8b2e8801c3c0 x1874672517869568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828183.445873:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517869568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828183.445888:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828183.445893:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828183.445897:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517869568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828183.445903:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517869696:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828183.445921:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828183.445931:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff8b2e8801d680 x1874672517869696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828183.445956:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517869696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828183.445966:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828183.445969:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828183.445973:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517869696:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828183.445980:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517869824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828183.445998:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828183.446008:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff8b2e8801de00 x1874672517869824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828183.446023:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517869824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828183.446033:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828183.446036:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828183.446039:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517869824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828183.446046:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517869952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828183.446063:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828183.446073:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828183/real 1787828183] req@ffff8b2e8801f0c0 x1874672517869952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828183.446087:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517869952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828208 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828183.446111:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828183.446117:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828183.446123:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517869952:192.168.203.101@tcp:8:kworker.0 00010000:00010000:0.0:1787828184.276799:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828184.276810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828184.468764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828185.300801:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828185.300852:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828185.492828:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828186.324786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828186.324849:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828186.516735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828187.348751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828187.348792:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828187.540738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:0.0:1787828188.372860:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828188.372890:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828188.564724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828188.564770:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828188.564776:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828188.564780:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828188.564788:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1194 00000100:00080000:1.0:1787828188.564797:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828188.564804:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828188.564835:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517870080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828188.564860:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828188.564864:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828188.564866:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828188.564871:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1194 00000100:00080000:1.0:1787828188.564876:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828188.564882:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828188.564891:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517870080:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787828188.564892:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828188.564914:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517870208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828188.564929:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828188.564932:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828188.564934:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828188.564936:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1194 00000100:00080000:1.0:1787828188.564948:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828188.564954:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828188.564955:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828188.564962:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828188.564974:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517870336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828188.564981:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828188.564984:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828188.564985:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828188.564988:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1194 00000100:00080000:1.0:1787828188.564991:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828188.564994:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff8b2e88077840 x1874672517870080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828188.564997:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828188.565003:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828188.565016:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517870464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828188.565019:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517870080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828188.565038:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828188.565041:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828188.565042:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828188.565043:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828188.565047:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828188.565051:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1194 00000100:00100000:0.0:1787828188.565055:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517870080:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828188.565056:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828188.565064:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828188.565067:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828188.565074:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517870592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828188.565080:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517870208:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828188.565084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828188.565087:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828188.565089:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828188.565092:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1194 00000100:00080000:1.0:1787828188.565095:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828188.565098:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828188.565102:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828188.565106:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828188.565117:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517870720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828188.565122:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff8b2e880752c0 x1874672517870208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828188.565132:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828188.565137:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828188.565141:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828188.565142:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517870208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828188.565149:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1194 00000100:00080000:1.0:1787828188.565155:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828188.565157:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828188.565160:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828188.565164:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787828188.565167:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828188.565169:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517870208:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828188.565178:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517870848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828188.565179:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517870336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828188.565200:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828188.565207:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff8b2e88075e00 x1874672517870336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828188.565226:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517870336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828188.565235:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828188.565238:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828188.565242:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517870336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828188.565248:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517870464:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828188.565267:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828188.565279:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff8b2e88076d00 x1874672517870464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828188.565287:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517870464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828188.565292:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828188.565294:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828188.565297:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517870464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828188.565308:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517870592:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828188.565321:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828188.565328:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff8b2e88074000 x1874672517870592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828188.565335:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517870592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828188.565345:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828188.565347:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828188.565350:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517870592:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828188.565353:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517870720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828188.565362:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828188.565368:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff8b2e880761c0 x1874672517870720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828188.565375:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517870720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828188.565380:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828188.565381:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828188.565383:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517870720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828188.565387:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517870848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828188.565395:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828188.565401:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828188/real 1787828188] req@ffff8b2e880770c0 x1874672517870848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828188.565407:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517870848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828213 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828188.565412:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828188.565413:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828188.565416:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517870848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828189.396787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828189.396916:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828189.588772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828190.420823:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828190.612797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828191.444724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828191.444749:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828191.636761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828192.468808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828192.468863:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828192.660784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828193.492743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:1.0:1787828193.684727:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828193.684737:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787828193.684739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828193.684740:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828193.684752:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1200 00000100:00080000:1.0:1787828193.684761:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828193.684768:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828193.684799:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517870976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828193.684821:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828193.684825:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828193.684827:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828193.684832:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1200 00000100:00080000:1.0:1787828193.684837:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828193.684842:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828193.684852:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828193.684856:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517870976:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828193.684868:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517871104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828193.684878:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828193.684880:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828193.684882:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828193.684884:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1200 00000100:00080000:1.0:1787828193.684888:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828193.684891:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828193.684895:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828193.684904:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517871232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828193.684911:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828193.684913:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828193.684914:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828193.684916:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1200 00000100:00080000:1.0:1787828193.684919:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828193.684921:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828193.684924:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828193.684931:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517871360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828193.684937:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828193.684939:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828193.684941:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828193.684945:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1200 00000100:00080000:1.0:1787828193.684946:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828193.684949:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828193.684952:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828193.684960:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517871104:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828193.684962:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517871488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828193.684978:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828193.684980:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828193.684981:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828193.684984:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1200 00000100:00080000:1.0:1787828193.684986:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828193.684988:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828193.684991:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828193.684999:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517871616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828193.685006:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828193.685008:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828193.685011:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828193.685012:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517871232:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787828193.685015:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1200 00000100:00080000:1.0:1787828193.685021:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828193.685024:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828193.685027:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828193.685036:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517871744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828193.685049:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517871360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828193.685084:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517871488:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828193.685101:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517871616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828193.685118:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517871744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828194.516758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828194.516794:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828194.708811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828195.540739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828195.540838:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828195.732746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828196.564784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828196.756763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:2.0:1787828196.756928:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787828196.756966:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828196.756981:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828196.757015:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828196.757022:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828196.757028:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828196.757035:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828196.757044:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828196.757049:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787828196.757080:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828196] req@ffff8b2e88074000 x1874672517871488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828196.757114:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517871488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828196.757136:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828196.757142:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828196.757147:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517871488:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828196.757156:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828196] req@ffff8b2e88076d00 x1874672517871616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828196.757176:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517871616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828196.757186:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828196.757190:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828196.757194:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517871616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828196.757203:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828196] req@ffff8b2e88075e00 x1874672517871744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828196.757217:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517871744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828196.757226:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828196.757229:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828196.757233:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517871744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828196.757242:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828196] req@ffff8b2e88074b40 x1874672517871104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828196.757255:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517871104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828196.757265:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828196.757269:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828196.757273:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517871104:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828196.757281:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828196] req@ffff8b2e880770c0 x1874672517871232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828196.757295:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517871232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828196.757305:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828196.757309:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828196.757313:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517871232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828196.757322:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828196] req@ffff8b2e880761c0 x1874672517871360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828196.757335:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517871360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828196.757344:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828196.757348:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828196.757352:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517871360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828196.757360:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828193/real 1787828196] req@ffff8b2e88076580 x1874672517870976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828196.757374:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517870976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828218 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828196.757383:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828196.757387:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828196.757396:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517870976:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828197.588742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828197.588792:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828197.780740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828198.612743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828198.612745:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828198.804730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828198.804770:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828198.804778:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828198.804784:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828198.804796:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1205 00000100:00080000:1.0:1787828198.804810:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828198.804819:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828198.804860:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517871872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828198.804888:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828198.804892:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828198.804894:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828198.804900:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1205 00000100:00080000:1.0:1787828198.804904:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828198.804909:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828198.804922:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517871872:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787828198.804922:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828198.804948:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517872000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828198.804961:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828198.804964:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828198.804967:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828198.804971:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1205 00000100:00080000:1.0:1787828198.804992:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828198.804996:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828198.805002:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1787828198.805008:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828198.805018:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517872128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828198.805030:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828198.805035:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828198.805037:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828198.805041:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1205 00000100:00080000:1.0:1787828198.805045:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828198.805049:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828198.805054:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828198.805067:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517872256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828198.805078:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff8b2e880752c0 x1874672517871872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828198.805085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828198.805093:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828198.805096:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828198.805104:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1205 00000100:00080000:1.0:1787828198.805109:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828198.805116:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880752c0 x1874672517871872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828198.805116:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828198.805122:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828198.805138:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828198.805140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517872384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828198.805148:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828198.805158:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828198.805160:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880752c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517871872:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828198.805163:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828198.805168:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828198.805175:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1205 00000100:00080000:1.0:1787828198.805178:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828198.805183:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828198.805189:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828198.805201:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517872512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828198.805202:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517872000:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828198.805226:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828198.805231:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828198.805233:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828198.805239:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1205 00000100:00080000:1.0:1787828198.805244:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828198.805248:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828198.805253:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828198.805259:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828198.805272:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff8b2e88077840 x1874672517872000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828198.805276:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517872640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828198.805314:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517872000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828198.805328:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828198.805333:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828198.805336:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517872000:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828198.805344:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517872128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828198.805369:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828198.805381:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff8b2e88074780 x1874672517872128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828198.805394:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517872128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828198.805403:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828198.805407:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828198.805410:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517872128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828198.805417:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517872256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828198.805439:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828198.805451:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff8b2e88076940 x1874672517872256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828198.805463:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517872256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828198.805472:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828198.805476:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828198.805479:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517872256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828198.805485:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517872384:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828198.805506:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828198.805517:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff8b2e88074f00 x1874672517872384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828198.805529:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517872384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828198.805537:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828198.805539:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828198.805542:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517872384:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828198.805559:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517872512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828198.805579:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828198.805589:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff8b2e88075680 x1874672517872512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828198.805601:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517872512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828198.805610:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828198.805613:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828198.805617:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517872512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828198.805623:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517872640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828198.805685:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828198.805699:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828198/real 1787828198] req@ffff8b2e88075a40 x1874672517872640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828198.805721:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517872640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828223 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828198.805730:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828198.805734:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828198.805737:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517872640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828199.636738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828199.828735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828200.660722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828200.852711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828201.684748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828201.684770:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828201.876733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828202.708751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828202.900750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828203.732735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828203.924724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828203.924761:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828203.924767:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828203.924771:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828203.924785:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1210 00000100:00080000:2.0:1787828203.924794:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828203.924809:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828203.924855:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517872768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828203.924883:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828203.924886:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828203.924887:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828203.924891:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1210 00000100:00080000:2.0:1787828203.924893:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828203.924896:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828203.924905:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828203.924929:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517872768:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828203.924929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517872896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828203.924944:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828203.924946:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828203.924948:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828203.924951:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1210 00000100:00080000:2.0:1787828203.924961:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828203.924964:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828203.924968:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828203.924978:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517873024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828203.924993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828203.924996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828203.924998:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828203.925001:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1210 00000100:00080000:2.0:1787828203.925003:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828203.925005:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828203.925009:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828203.925025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517873152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828203.925036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828203.925038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828203.925039:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828203.925042:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1210 00000100:00080000:2.0:1787828203.925044:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828203.925047:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828203.925050:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828203.925060:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787828203.925069:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517873280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828203.925084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828203.925087:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828203.925089:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828203.925092:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1210 00000100:00080000:2.0:1787828203.925094:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828203.925096:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828203.925100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828203.925118:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517873408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828203.925120:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff8b2e8801f0c0 x1874672517872768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828203.925133:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828203.925136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828203.925137:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828203.925147:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1210 00000100:00080000:2.0:1787828203.925151:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828203.925154:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517872768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828203.925157:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828203.925162:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828203.925169:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517873536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828203.925206:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828203.925211:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828203.925214:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517872768:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828203.925245:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517872896:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828203.925263:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828203.925273:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff8b2e8801de00 x1874672517872896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828203.925281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517872896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828203.925287:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828203.925290:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828203.925292:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517872896:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828203.925296:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517873024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828203.925307:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828203.925313:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff8b2e8801d680 x1874672517873024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828203.925321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517873024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828203.925326:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828203.925328:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828203.925330:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517873024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828203.925333:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517873152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828203.925342:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828203.925347:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff8b2e8801c3c0 x1874672517873152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828203.925354:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517873152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828203.925360:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828203.925362:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828203.925364:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517873152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828203.925367:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517873280:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828203.925376:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828203.925381:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff8b2e8801cb40 x1874672517873280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828203.925388:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517873280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828203.925394:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828203.925396:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828203.925398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517873280:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828203.925401:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517873408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828203.925410:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828203.925415:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff8b2e8801ed00 x1874672517873408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828203.925422:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517873408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828203.925445:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828203.925448:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828203.925451:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517873408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828203.925455:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517873536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828203.925480:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828203.925499:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828203/real 1787828203] req@ffff8b2e8801c780 x1874672517873536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828203.925507:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517873536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828228 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828203.925512:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828203.925527:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828203.925529:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517873536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828204.756726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828204.756755:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828204.948734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828205.780739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828205.972726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828206.804737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828206.996718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828207.828748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828207.828784:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828208.020704:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828208.852731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828209.044750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828209.044797:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828209.044804:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828209.044808:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828209.044819:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1215 00000100:00080000:2.0:1787828209.044830:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828209.044838:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828209.044876:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517873664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828209.044906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828209.044910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828209.044912:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828209.044917:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1215 00000100:00080000:2.0:1787828209.044921:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828209.044926:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828209.044937:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828209.044944:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517873664:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828209.044956:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517873792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828209.044968:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828209.044972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828209.044974:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828209.044978:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1215 00000100:00080000:2.0:1787828209.044982:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828209.044987:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828209.044993:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828209.045006:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517873920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828209.045018:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828209.045019:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828209.045025:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828209.045028:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828209.045033:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1215 00000100:00080000:2.0:1787828209.045038:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828209.045042:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828209.045048:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828209.045061:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517874048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828209.045068:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828209/real 1787828209] req@ffff8b2e8801e580 x1874672517873664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828209.045076:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828209.045085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828209.045088:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828209.045095:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1215 00000100:00080000:2.0:1787828209.045099:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828209.045102:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517873664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828209.045107:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828209.045116:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828209.045121:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828209.045130:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787828209.045132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517874176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828209.045138:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517873664:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787828209.045152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828209.045156:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828209.045158:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828209.045163:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1215 00000100:00080000:2.0:1787828209.045168:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828209.045170:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517873792:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828209.045175:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828209.045183:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828209.045199:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517874304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828209.045204:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828209.045217:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828209.045222:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828209/real 1787828209] req@ffff8b2e8801e940 x1874672517873792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828209.045224:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828209.045228:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828209.045237:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1215 00000100:00080000:2.0:1787828209.045244:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828209.045248:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517873792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828209.045252:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828209.045261:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828209.045265:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828209.045271:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828209.045276:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517873792:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787828209.045279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517874432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828209.045287:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517873920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828209.045315:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828209.045328:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828209/real 1787828209] req@ffff8b2e8801f480 x1874672517873920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828209.045342:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517873920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828209.045350:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828209.045354:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828209.045357:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517873920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828209.045364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517874048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828209.045384:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828209.045395:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828209/real 1787828209] req@ffff8b2e8801fc00 x1874672517874048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828209.045408:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517874048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828209.045416:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828209.045420:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828209.045423:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517874048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828209.045430:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517874176:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828209.045446:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828209.045456:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828209/real 1787828209] req@ffff8b2e8801cf00 x1874672517874176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828209.045467:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672517874176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828209.045477:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828209.045481:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828209.045484:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517874176:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828209.045499:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517874304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828209.045516:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828209.045526:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828209/real 1787828209] req@ffff8b2e8801c000 x1874672517874304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828209.045545:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672517874304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828209.045553:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828209.045555:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828209.045559:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517874304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828209.045578:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517874432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828209.045597:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828209.045607:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828209/real 1787828209] req@ffff8b2e8801da40 x1874672517874432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828209.045619:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672517874432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828234 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828209.045627:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828209.045631:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828209.045634:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517874432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828209.876738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828210.068745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828210.900730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828210.900752:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828211.092722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828211.924778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828212.116779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828212.948789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828213.140748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828213.972724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828213.972853:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828214.164764:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828214.164768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828214.164776:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828214.164782:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828214.164794:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1220 00000100:00080000:0.0:1787828214.164806:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828214.164814:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828214.164859:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517874560/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828214.164885:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828214.164889:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828214.164890:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828214.164894:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1220 00000100:00080000:0.0:1787828214.164896:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828214.164900:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828214.164909:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828214.164922:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517874688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828214.164933:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828214.164936:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828214.164937:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828214.164939:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1220 00000100:00080000:0.0:1787828214.164941:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828214.164943:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828214.164947:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828214.164955:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517874816/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828214.164966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828214.164969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828214.164970:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828214.164972:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1220 00000100:00080000:0.0:1787828214.164975:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828214.164977:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828214.164980:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828214.164987:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517874944/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828214.164992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828214.164995:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828214.164996:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828214.164999:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1220 00000100:00080000:0.0:1787828214.165003:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828214.165009:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828214.165013:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828214.165021:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517875072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828214.165027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828214.165029:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828214.165030:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828214.165033:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1220 00000100:00080000:0.0:1787828214.165036:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828214.165039:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828214.165043:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828214.165050:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517875200/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828214.165056:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828214.165058:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828214.165059:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828214.165063:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1220 00000100:00080000:0.0:1787828214.165066:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828214.165072:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828214.165078:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828214.165092:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517875328/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828214.996786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828215.188733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828216.020727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828216.212757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828216.212788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828217.044782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828217.044791:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828217.236694:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828217.236734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828218.068808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787828218.260787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828218.260817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828219.092757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828219.284727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828219.284752:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828219.284758:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828219.284762:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828219.284770:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1225 00000100:00080000:1.0:1787828219.284781:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828219.284788:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828219.284822:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517875456/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828219.284849:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828219.284853:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828219.284855:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828219.284861:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1225 00000100:00080000:1.0:1787828219.284865:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828219.284871:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828219.284882:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828219.284900:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517875584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828219.284910:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828219.284913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828219.284914:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828219.284918:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1225 00000100:00080000:1.0:1787828219.284921:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828219.284924:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828219.284928:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828219.284942:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517875712/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828219.284950:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828219.284953:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828219.284954:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828219.284957:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1225 00000100:00080000:1.0:1787828219.284959:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828219.284962:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828219.284966:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828219.284975:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517875840/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828219.284996:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828219.285000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828219.285001:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828219.285006:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1225 00000100:00080000:1.0:1787828219.285012:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828219.285020:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828219.285030:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828219.285048:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517875968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828219.285069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828219.285072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828219.285074:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828219.285081:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1225 00000100:00080000:1.0:1787828219.285088:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828219.285092:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828219.285098:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828219.285115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517876096/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828219.285135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828219.285140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828219.285145:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828219.285155:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1225 00000100:00080000:1.0:1787828219.285159:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828219.285167:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828219.285172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828219.285188:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517876224/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828220.116775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828220.116813:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828220.308748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828221.140726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828221.332729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828222.164771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828222.356763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828223.188769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828223.188809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828223.380752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828224.212778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:1.0:1787828224.404745:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828224.404746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828224.404757:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828224.404762:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828224.404771:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1230 00000100:00080000:1.0:1787828224.404783:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828224.404790:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828224.404824:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517876352/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828224.404846:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828224.404848:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828224.404850:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828224.404853:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1230 00000100:00080000:1.0:1787828224.404855:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828224.404858:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828224.404866:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828224.404878:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517876480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828224.404895:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828224.404898:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828224.404899:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828224.404902:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1230 00000100:00080000:1.0:1787828224.404904:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828224.404906:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828224.404910:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828224.404917:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517876608/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828224.404923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828224.404925:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828224.404926:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828224.404929:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1230 00000100:00080000:1.0:1787828224.404932:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828224.404934:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828224.404936:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828224.404954:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517876736/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828224.404972:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828224.404975:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828224.404977:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828224.404980:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1230 00000100:00080000:1.0:1787828224.404983:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828224.404986:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828224.404989:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828224.404997:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517876864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828224.405003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828224.405007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828224.405010:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828224.405016:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1230 00000100:00080000:1.0:1787828224.405020:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828224.405025:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828224.405029:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828224.405036:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517876992/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828224.405043:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828224.405046:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828224.405049:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828224.405057:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1230 00000100:00080000:1.0:1787828224.405062:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828224.405067:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828224.405072:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828224.405080:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517877120/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828225.236723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828225.428724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828226.260727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828226.260819:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828226.452734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828227.284818:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828227.284921:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828227.476756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828228.308721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828228.500720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828229.332718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828229.332798:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828229.524705:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828229.524733:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828229.524740:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828229.524745:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828229.524753:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1235 00000100:00080000:1.0:1787828229.524762:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828229.524768:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828229.524799:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517877248/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828229.524820:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828229.524830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828229.524831:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828229.524834:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1235 00000100:00080000:1.0:1787828229.524837:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828229.524840:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828229.524849:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828229.524860:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517877376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828229.524869:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828229.524872:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828229.524873:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828229.524877:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1235 00000100:00080000:1.0:1787828229.524879:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828229.524881:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828229.524885:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828229.524894:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517877504/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828229.524900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828229.524902:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828229.524904:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828229.524906:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1235 00000100:00080000:1.0:1787828229.524909:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828229.524910:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828229.524913:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828229.524924:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517877632/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828229.524932:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828229.524934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828229.524936:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828229.524939:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1235 00000100:00080000:1.0:1787828229.524941:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828229.524943:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828229.524946:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828229.524954:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517877760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828229.524970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828229.524976:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828229.524977:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828229.524982:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1235 00000100:00080000:1.0:1787828229.524985:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828229.524992:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828229.524998:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828229.525008:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517877888/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828229.525025:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828229.525028:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828229.525029:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828229.525032:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1235 00000100:00080000:1.0:1787828229.525034:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828229.525036:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828229.525039:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828229.525049:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517878016/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828230.356732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828230.356783:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828230.548714:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828231.380731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828231.380756:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828231.572709:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828232.404708:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828232.404824:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828232.596711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828233.428702:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828233.428779:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828233.620711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828234.452735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828234.452891:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828234.644781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828234.644797:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828234.644806:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787828234.644810:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828234.644811:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828234.644824:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1241 00000100:00080000:1.0:1787828234.644853:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828234.644862:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828234.644920:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517878144/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828234.644957:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828234.644962:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828234.644980:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828234.644984:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1241 00000100:00080000:1.0:1787828234.644988:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828234.644993:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828234.645003:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828234.645020:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517878272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828234.645032:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828234.645036:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828234.645038:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828234.645042:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1241 00000100:00080000:1.0:1787828234.645047:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828234.645051:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828234.645056:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828234.645071:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517878400/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828234.645083:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828234.645087:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828234.645089:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828234.645092:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1241 00000100:00080000:1.0:1787828234.645096:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828234.645101:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828234.645106:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828234.645125:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517878528/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828234.645160:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828234.645168:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828234.645173:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828234.645182:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1241 00000100:00080000:1.0:1787828234.645186:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828234.645191:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828234.645199:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828234.645218:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517878656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828234.645234:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828234.645241:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828234.645244:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828234.645255:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1241 00000100:00080000:1.0:1787828234.645262:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828234.645272:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828234.645282:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828234.645301:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517878784/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828234.645333:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828234.645343:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828234.645347:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828234.645354:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1241 00000100:00080000:1.0:1787828234.645359:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828234.645367:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828234.645376:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828234.645410:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517878912/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828235.476791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828235.476902:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828235.668781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828235.668792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828236.500786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828236.500856:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828236.692711:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828236.692715:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828237.524717:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828237.524794:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828237.716742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828237.716748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828238.548828:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828238.548933:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828238.740865:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828239.572787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828239.764743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828239.764745:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.764758:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828239.764763:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.764772:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1246 00000100:00080000:1.0:1787828239.764782:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.764789:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828239.764834:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517879040/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828239.764864:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.764868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828239.764871:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.764876:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1246 00000100:00080000:1.0:1787828239.764881:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.764885:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828239.764895:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828239.764911:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517879168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828239.764929:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.764933:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828239.764934:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.764937:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1246 00000100:00080000:1.0:1787828239.764942:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.764945:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828239.764948:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828239.764960:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517879296/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828239.764974:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.764982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828239.764984:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.764989:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1246 00000100:00080000:1.0:1787828239.764992:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.764996:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828239.765000:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828239.765013:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517879424/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828239.765039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.765042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828239.765046:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.765056:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1246 00000100:00080000:1.0:1787828239.765062:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.765075:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828239.765085:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828239.765102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517879552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828239.765113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.765117:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828239.765121:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.765126:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1246 00000100:00080000:1.0:1787828239.765129:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.765132:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828239.765136:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828239.765149:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517879680/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828239.765158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828239.765162:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828239.765166:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828239.765175:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1246 00000100:00080000:1.0:1787828239.765182:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828239.765188:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828239.765195:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828239.765211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517879808/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828240.596802:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828240.596899:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828240.788837:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828241.620817:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828241.620940:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828241.812841:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828242.644820:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828242.836727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828243.668754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828243.668855:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828243.860747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828244.692760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828244.692891:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828244.884755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828244.884823:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828244.884830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828244.884835:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.884846:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1251 00000100:00080000:0.0:1787828244.884859:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.884868:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828244.884927:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517879936/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828244.884958:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828244.884962:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828244.884982:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.884987:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1251 00000100:00080000:0.0:1787828244.884992:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.884998:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828244.885011:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828244.885027:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517880064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828244.885055:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828244.885060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828244.885062:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.885068:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1251 00000100:00080000:0.0:1787828244.885074:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.885078:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828244.885100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828244.885115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517880192/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828244.885124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828244.885131:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828244.885137:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.885141:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1251 00000100:00080000:0.0:1787828244.885146:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.885150:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828244.885154:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828244.885165:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517880320/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828244.885178:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828244.885181:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828244.885183:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.885186:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1251 00000100:00080000:0.0:1787828244.885191:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.885197:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828244.885224:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828244.885244:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517880448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828244.885257:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828244.885263:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828244.885270:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.885284:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1251 00000100:00080000:0.0:1787828244.885293:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.885300:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828244.885311:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828244.885364:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517880576/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828244.885372:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828244.885375:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828244.885377:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828244.885380:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1251 00000100:00080000:0.0:1787828244.885384:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828244.885390:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828244.885395:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828244.885410:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517880704/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828245.716826:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828245.908852:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828246.740840:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828246.740947:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828246.932748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828247.764874:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828247.765021:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828247.956894:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828248.788739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828248.980867:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828249.812726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828249.812773:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828250.004726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828250.004753:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828250.004761:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828250.004765:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828250.004775:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1256 00000100:00080000:1.0:1787828250.004786:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828250.004793:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828250.004828:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517880832/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828250.004855:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828250.004859:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828250.004861:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828250.004867:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1256 00000100:00080000:1.0:1787828250.004872:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828250.004876:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828250.004887:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828250.004903:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517880960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828250.004915:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828250.004919:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828250.004921:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828250.004925:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1256 00000100:00080000:1.0:1787828250.004931:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828250.004935:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828250.004941:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828250.004965:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517881088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828250.004976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828250.004981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828250.004986:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828250.004994:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1256 00000100:00080000:1.0:1787828250.004998:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828250.005004:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828250.005013:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828250.005034:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517881216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828250.005047:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828250.005051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828250.005053:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828250.005061:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1256 00000100:00080000:1.0:1787828250.005066:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828250.005071:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828250.005077:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828250.005093:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517881344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828250.005117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828250.005124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828250.005129:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828250.005140:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1256 00000100:00080000:1.0:1787828250.005147:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828250.005156:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828250.005166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828250.005186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517881472/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828250.005202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828250.005212:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828250.005218:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828250.005226:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1256 00000100:00080000:1.0:1787828250.005236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828250.005246:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828250.005255:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828250.005274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517881600/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828250.836703:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828250.836775:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828251.028699:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828251.860742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828252.052754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828252.884703:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828252.884778:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828253.076745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828253.908718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828253.908775:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828254.100779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828254.932784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828255.124745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828255.124749:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828255.124756:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828255.124760:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828255.124769:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1261 00000100:00080000:1.0:1787828255.124805:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828255.124812:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828255.124848:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517881728/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828255.124872:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828255.124876:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828255.124879:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828255.124884:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1261 00000100:00080000:1.0:1787828255.124889:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828255.124895:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828255.124908:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828255.124929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517881856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828255.124944:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828255.124947:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828255.124949:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828255.124953:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1261 00000100:00080000:1.0:1787828255.124957:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828255.124960:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828255.124965:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828255.124979:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517881984/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828255.124990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828255.124992:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828255.124994:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828255.124997:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1261 00000100:00080000:1.0:1787828255.125000:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828255.125003:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828255.125006:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828255.125032:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517882112/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828255.125040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828255.125046:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828255.125050:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828255.125059:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1261 00000100:00080000:1.0:1787828255.125064:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828255.125068:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828255.125089:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828255.125107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517882240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828255.125126:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828255.125133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828255.125136:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828255.125141:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1261 00000100:00080000:1.0:1787828255.125144:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828255.125147:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828255.125150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828255.125161:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517882368/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828255.125169:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828255.125171:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828255.125173:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828255.125176:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1261 00000100:00080000:1.0:1787828255.125178:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828255.125181:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828255.125184:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828255.125194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517882496/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828255.956735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828255.956801:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828256.148709:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828256.980723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828256.980783:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828257.172785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828258.004775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828258.004956:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828258.196793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828259.028731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828259.028792:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828259.220771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828259.220806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828260.052718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828260.052814:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828260.244728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828260.244730:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828260.244737:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828260.244741:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828260.244750:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1266 00000100:00080000:2.0:1787828260.244759:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828260.244766:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828260.244802:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517882624/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828260.244823:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828260.244826:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828260.244828:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828260.244831:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1266 00000100:00080000:2.0:1787828260.244833:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828260.244836:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828260.244846:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828260.244858:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517882752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828260.244866:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828260.244869:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828260.244871:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828260.244873:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1266 00000100:00080000:2.0:1787828260.244875:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828260.244877:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828260.244881:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828260.244889:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517882880/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828260.244895:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828260.244897:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828260.244899:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828260.244901:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1266 00000100:00080000:2.0:1787828260.244904:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828260.244906:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828260.244909:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828260.244916:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517883008/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828260.244922:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828260.244925:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828260.244926:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828260.244930:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1266 00000100:00080000:2.0:1787828260.244932:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828260.244935:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828260.244939:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828260.244958:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517883136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828260.244967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828260.244974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828260.244976:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828260.244980:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1266 00000100:00080000:2.0:1787828260.244984:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828260.244986:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828260.244990:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828260.245000:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517883264/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828260.245006:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828260.245008:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828260.245010:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828260.245026:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1266 00000100:00080000:2.0:1787828260.245039:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828260.245045:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828260.245049:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828260.245059:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517883392/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828261.076751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828261.076799:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828261.268850:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828262.100733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828262.100820:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828263.124806:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828263.124890:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828263.317138:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:2.0:1787828263.317319:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787828263.317397:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828263.317436:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828263.317509:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828263.317539:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828263.317555:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828263.317570:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828263.317592:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828263.317616:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:2.0:1787828264.148768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828264.148902:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828264.340874:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828265.172774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828265.172876:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828265.364842:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828265.364861:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828265.364861:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828265.364870:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828265.364888:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1271 00000100:00080000:1.0:1787828265.364906:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828265.364939:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828265.364993:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517883520/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828265.365032:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828265.365039:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828265.365043:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828265.365052:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1271 00000100:00080000:1.0:1787828265.365059:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828265.365067:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828265.365084:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828265.365110:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517883648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828265.365129:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828265.365135:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828265.365139:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828265.365145:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1271 00000100:00080000:1.0:1787828265.365151:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828265.365157:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828265.365165:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828265.365213:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517883776/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828265.365242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828265.365249:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828265.365252:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828265.365267:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1271 00000100:00080000:1.0:1787828265.365274:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828265.365280:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828265.365289:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828265.365308:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517883904/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828265.365326:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828265.365333:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828265.365337:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828265.365345:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1271 00000100:00080000:1.0:1787828265.365351:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828265.365358:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828265.365367:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828265.365384:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517884032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828265.365401:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828265.365407:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828265.365410:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828265.365418:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1271 00000100:00080000:1.0:1787828265.365423:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828265.365430:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828265.365438:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828265.365457:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517884160/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828265.365476:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828265.365483:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828265.365486:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828265.365493:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1271 00000100:00080000:1.0:1787828265.365500:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828265.365507:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828265.365515:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828265.365533:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517884288/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828266.196802:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828266.196926:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828266.388919:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828267.220712:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828267.220815:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828267.412731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828268.244712:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828268.244822:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828268.436739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828269.268714:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828269.268836:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828269.460745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828270.292813:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828270.293056:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828270.484765:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828270.484793:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828270.484801:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828270.484819:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1276 00000100:00080000:1.0:1787828270.484839:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828270.484852:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828270.484879:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828270.484930:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517884416/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828270.484992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828270.484997:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828270.484999:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828270.485005:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1276 00000100:00080000:1.0:1787828270.485009:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828270.485013:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828270.485027:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828270.485042:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517884544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828270.485054:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828270.485060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828270.485067:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828270.485077:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1276 00000100:00080000:1.0:1787828270.485092:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828270.485095:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828270.485100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828270.485117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517884672/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828270.485127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828270.485131:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828270.485132:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828270.485153:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1276 00000100:00080000:1.0:1787828270.485158:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828270.485163:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828270.485168:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828270.485181:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517884800/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828270.485191:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828270.485198:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828270.485203:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828270.485213:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1276 00000100:00080000:1.0:1787828270.485217:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828270.485222:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828270.485227:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828270.485241:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517884928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828270.485251:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828270.485254:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828270.485256:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828270.485261:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1276 00000100:00080000:1.0:1787828270.485266:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828270.485270:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828270.485275:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828270.485291:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517885056/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828270.485304:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828270.485309:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828270.485313:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828270.485324:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1276 00000100:00080000:1.0:1787828270.485333:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828270.485342:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828270.485348:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828270.485367:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517885184/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828271.316803:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828271.316989:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828271.508796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828272.340849:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828272.340939:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828272.532886:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828273.364817:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828273.364978:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828273.556853:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828274.388808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828274.388917:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828274.580814:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828275.412784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828275.412829:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828275.604774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828275.604774:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.604783:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828275.604788:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.604796:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1282 00000100:00080000:2.0:1787828275.604808:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.604815:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828275.604849:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517885312/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828275.604869:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.604872:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828275.604873:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.604876:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1282 00000100:00080000:2.0:1787828275.604879:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.604882:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828275.604897:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828275.604912:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517885440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828275.604921:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.604924:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828275.604925:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.604928:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1282 00000100:00080000:2.0:1787828275.604931:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.604933:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828275.604937:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828275.604944:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517885568/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828275.604952:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.604954:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828275.604956:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.604962:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1282 00000100:00080000:2.0:1787828275.604967:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.604969:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828275.604973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828275.604982:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517885696/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828275.604988:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.604991:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828275.604992:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.604995:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1282 00000100:00080000:2.0:1787828275.604998:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.605003:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828275.605010:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828275.605022:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517885824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828275.605029:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.605034:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828275.605036:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.605042:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1282 00000100:00080000:2.0:1787828275.605047:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.605056:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828275.605067:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828275.605084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517885952/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828275.605117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828275.605120:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828275.605123:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828275.605141:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1282 00000100:00080000:2.0:1787828275.605147:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828275.605157:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828275.605165:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828275.605178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517886080/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828276.436759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828276.436805:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828276.628754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828277.460731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828277.460768:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828277.652748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828278.484715:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828278.484792:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828278.676742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828279.508749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828279.700745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828280.532704:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828280.532763:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828280.724742:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828280.724747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828280.724754:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828280.724759:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.724768:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1287 00000100:00080000:2.0:1787828280.724778:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.724784:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828280.724819:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517886208/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828280.724845:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.724848:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828280.724849:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.724852:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1287 00000100:00080000:2.0:1787828280.724855:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.724858:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828280.724875:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828280.724887:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517886336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828280.724895:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.724898:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828280.724900:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.724902:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1287 00000100:00080000:2.0:1787828280.724905:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.724907:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828280.724911:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828280.724918:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517886464/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828280.724925:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.724927:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828280.724928:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.724930:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1287 00000100:00080000:2.0:1787828280.724933:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.724940:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828280.724944:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828280.724952:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517886592/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828280.724958:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.724960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828280.724962:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.724964:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1287 00000100:00080000:2.0:1787828280.724967:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.724969:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828280.724973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828280.724982:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517886720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828280.725012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.725015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828280.725016:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.725022:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1287 00000100:00080000:2.0:1787828280.725027:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.725032:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828280.725039:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828280.725048:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517886848/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828280.725085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828280.725088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828280.725090:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828280.725094:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1287 00000100:00080000:2.0:1787828280.725097:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828280.725100:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828280.725104:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828280.725115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517886976/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828281.556809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828281.556809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828281.748752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828282.580804:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828282.772771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828283.604773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828283.604891:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828283.796758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828283.796959:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828284.628761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828284.628839:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828284.820735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828285.652799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828285.844764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828285.844858:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828285.844871:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828285.844876:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828285.844893:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1292 00000100:00080000:2.0:1787828285.844907:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828285.844916:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828285.844970:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517887104/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828285.844991:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828285.844994:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828285.844996:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828285.844999:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1292 00000100:00080000:2.0:1787828285.845003:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828285.845006:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828285.845016:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828285.845044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517887232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828285.845090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828285.845096:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828285.845100:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828285.845108:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1292 00000100:00080000:2.0:1787828285.845114:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828285.845120:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828285.845128:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828285.845146:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517887360/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828285.845159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828285.845177:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828285.845182:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828285.845190:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1292 00000100:00080000:2.0:1787828285.845198:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828285.845203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828285.845210:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828285.845226:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517887488/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828285.845238:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828285.845248:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828285.845256:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828285.845290:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1292 00000100:00080000:2.0:1787828285.845299:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828285.845313:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828285.845330:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828285.845350:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517887616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828285.845398:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828285.845413:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828285.845422:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828285.845447:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1292 00000100:00080000:2.0:1787828285.845467:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828285.845486:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828285.845505:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828285.845547:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517887744/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828285.845606:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828285.845621:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828285.845631:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828285.845642:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1292 00000100:00080000:2.0:1787828285.845648:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828285.845653:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828285.845658:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828285.845668:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517887872/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828286.676800:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828286.676858:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828286.868731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828287.700753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828287.700826:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828287.892726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828288.724754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828288.916781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828289.748805:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828289.748863:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828289.940718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828290.772744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828290.964720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828290.964723:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.964730:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828290.964734:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.964743:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1297 00000100:00080000:2.0:1787828290.964753:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.964760:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828290.964793:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517888000/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.964824:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.964827:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828290.964828:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.964831:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1297 00000100:00080000:2.0:1787828290.964834:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.964837:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828290.964846:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828290.964858:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517888128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.964869:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.964871:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828290.964873:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.964875:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1297 00000100:00080000:2.0:1787828290.964878:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.964880:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828290.964883:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828290.964891:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517888256/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.964897:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.964900:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828290.964901:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.964904:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1297 00000100:00080000:2.0:1787828290.964907:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.964909:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828290.964912:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828290.964923:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517888384/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.964930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.964933:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828290.964934:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.964937:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1297 00000100:00080000:2.0:1787828290.964939:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.964941:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828290.964944:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828290.964952:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517888512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.964964:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.964972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828290.964975:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.964979:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1297 00000100:00080000:2.0:1787828290.964981:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.964986:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828290.965004:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828290.965015:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517888640/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828290.965034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828290.965037:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828290.965038:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828290.965042:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1297 00000100:00080000:2.0:1787828290.965044:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828290.965047:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828290.965050:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828290.965084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517888768/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828291.796719:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828291.988726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828292.820738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828292.820830:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828293.012714:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828293.844738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828293.844797:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828294.036701:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828294.868756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828295.060703:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828295.892733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828295.892807:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828296.084755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828296.084761:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828296.084770:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828296.084776:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828296.084796:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1302 00000100:00080000:2.0:1787828296.084812:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828296.084821:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828296.084873:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517888896/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828296.084901:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828296.084904:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828296.084905:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828296.084909:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1302 00000100:00080000:2.0:1787828296.084912:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828296.084916:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828296.084927:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828296.084942:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517889024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828296.084955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828296.084971:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828296.084973:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828296.084977:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1302 00000100:00080000:2.0:1787828296.084980:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828296.084983:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828296.084987:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828296.084998:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517889152/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828296.085006:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828296.085009:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828296.085010:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828296.085012:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1302 00000100:00080000:2.0:1787828296.085015:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828296.085017:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828296.085020:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828296.085027:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517889280/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828296.085034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828296.085038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828296.085043:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828296.085047:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1302 00000100:00080000:2.0:1787828296.085049:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828296.085052:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828296.085055:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828296.085062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517889408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828296.085068:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828296.085070:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828296.085071:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828296.085073:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1302 00000100:00080000:2.0:1787828296.085076:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828296.085078:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828296.085082:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828296.085091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517889536/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828296.085099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828296.085106:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828296.085109:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828296.085117:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1302 00000100:00080000:2.0:1787828296.085123:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828296.085126:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828296.085129:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828296.085138:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517889664/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828296.916777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828296.916845:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828297.108766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828297.940810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828298.132773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828298.964763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828298.964793:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828299.156744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828299.988762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828299.988828:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828300.180718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828301.012737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828301.012797:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828301.204787:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828301.204798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828301.204798:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828301.204806:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828301.204819:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1307 00000100:00080000:2.0:1787828301.204831:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828301.204839:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828301.204879:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517889792/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828301.204904:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828301.204909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828301.204911:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828301.204915:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1307 00000100:00080000:2.0:1787828301.204919:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828301.204936:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828301.204945:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828301.204959:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517889920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828301.204970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828301.204973:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828301.204974:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828301.204977:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1307 00000100:00080000:2.0:1787828301.204980:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828301.204983:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828301.204987:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828301.204995:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517890048/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828301.205002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828301.205004:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828301.205006:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828301.205008:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1307 00000100:00080000:2.0:1787828301.205011:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828301.205017:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828301.205022:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828301.205029:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517890176/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828301.205036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828301.205038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828301.205039:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828301.205042:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1307 00000100:00080000:2.0:1787828301.205044:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828301.205047:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828301.205051:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828301.205060:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517890304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828301.205087:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828301.205095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828301.205096:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828301.205100:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1307 00000100:00080000:2.0:1787828301.205103:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828301.205108:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828301.205114:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828301.205124:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517890432/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828301.205140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828301.205144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828301.205146:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828301.205149:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1307 00000100:00080000:2.0:1787828301.205151:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828301.205154:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828301.205156:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828301.205165:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517890560/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828302.036725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828302.036785:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828302.228757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828303.060740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787828303.252744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828304.084766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828304.084849:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828304.276784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828305.108742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828305.108805:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828305.300731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828306.132783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787828306.324740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828306.324740:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.324753:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828306.324760:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.324772:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1312 00000100:00080000:2.0:1787828306.324788:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.324797:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828306.324840:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517890688/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.324865:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.324868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828306.324870:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.324873:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1312 00000100:00080000:2.0:1787828306.324875:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.324878:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828306.324905:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828306.324922:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517890816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.324932:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.324935:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828306.324936:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.324939:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1312 00000100:00080000:2.0:1787828306.324942:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.324945:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828306.324949:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828306.324957:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517890944/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.324969:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.324972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828306.324973:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.324976:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1312 00000100:00080000:2.0:1787828306.324985:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.324988:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828306.324991:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828306.324999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517891072/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.325009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.325014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828306.325015:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.325019:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1312 00000100:00080000:2.0:1787828306.325022:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.325024:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828306.325027:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828306.325034:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517891200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.325041:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.325043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828306.325044:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.325047:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1312 00000100:00080000:2.0:1787828306.325050:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.325053:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828306.325056:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828306.325065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517891328/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828306.325071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828306.325074:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828306.325077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828306.325084:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1312 00000100:00080000:2.0:1787828306.325088:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828306.325094:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828306.325099:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828306.325107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517891456/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828307.156762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828307.156827:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828307.348741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828308.180766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828308.180803:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828308.372717:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828309.204742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787828309.396710:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828310.228810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828310.228850:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828310.420735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828311.252754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828311.252870:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828311.444726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828311.444730:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828311.444737:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828311.444740:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828311.444749:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1317 00000100:00080000:2.0:1787828311.444758:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828311.444765:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828311.444799:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517891584/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828311.444820:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828311.444823:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828311.444824:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828311.444827:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1317 00000100:00080000:2.0:1787828311.444829:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828311.444832:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828311.444839:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828311.444849:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517891712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828311.444874:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828311.444878:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828311.444879:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828311.444883:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1317 00000100:00080000:2.0:1787828311.444886:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828311.444888:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828311.444892:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828311.444902:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517891840/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828311.444909:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828311.444911:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828311.444912:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828311.444915:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1317 00000100:00080000:2.0:1787828311.444918:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828311.444921:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828311.444924:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828311.444933:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517891968/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828311.444941:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828311.444944:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828311.444945:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828311.444949:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1317 00000100:00080000:2.0:1787828311.444951:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828311.444953:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828311.444956:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828311.444963:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517892096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828311.444969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828311.444974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828311.444977:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828311.444984:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1317 00000100:00080000:2.0:1787828311.444989:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828311.444995:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828311.444999:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828311.445008:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517892224/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828311.445036:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828311.445044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828311.445049:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828311.445058:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1317 00000100:00080000:2.0:1787828311.445062:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828311.445066:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828311.445072:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828311.445092:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517892352/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828312.276778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787828312.468728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828313.300719:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828313.300760:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828313.492750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828314.324759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828314.324845:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828314.516707:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828315.348758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787828315.540753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828316.372814:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828316.372841:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828316.564841:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828316.564916:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828316.564942:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828316.564949:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828316.564962:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1322 00000100:00080000:0.0:1787828316.564977:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828316.565033:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828316.565103:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517892480/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828316.565131:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828316.565151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828316.565153:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828316.565158:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1322 00000100:00080000:0.0:1787828316.565166:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828316.565170:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828316.565181:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828316.565195:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517892608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828316.565203:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828316.565207:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828316.565208:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828316.565211:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1322 00000100:00080000:0.0:1787828316.565213:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828316.565216:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828316.565221:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828316.565229:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517892736/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828316.565236:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828316.565239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828316.565240:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828316.565243:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1322 00000100:00080000:0.0:1787828316.565248:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828316.565250:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828316.565253:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828316.565263:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517892864/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828316.565271:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828316.565274:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828316.565293:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828316.565299:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1322 00000100:00080000:0.0:1787828316.565314:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828316.565319:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828316.565325:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828316.565354:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517892992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828316.565398:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828316.565403:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828316.565406:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828316.565415:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1322 00000100:00080000:0.0:1787828316.565423:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828316.565432:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828316.565440:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828316.565453:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517893120/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828316.565490:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828316.565492:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828316.565494:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828316.565497:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1322 00000100:00080000:0.0:1787828316.565499:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828316.565502:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828316.565506:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828316.565516:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517893248/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828317.396772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828317.396963:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828317.588785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828318.420721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787828318.612850:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828319.444749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828319.444836:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828319.636764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828320.468730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828320.468893:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828320.660721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828321.492713:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828321.492804:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828321.684822:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828321.684831:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828321.684840:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828321.684846:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828321.684858:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1328 00000100:00080000:1.0:1787828321.684870:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828321.684888:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828321.684928:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517893376/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828321.684975:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828321.684980:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828321.684982:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828321.684987:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1328 00000100:00080000:1.0:1787828321.684992:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828321.684998:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828321.685012:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828321.685043:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517893504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828321.685068:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828321.685072:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828321.685075:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828321.685080:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1328 00000100:00080000:1.0:1787828321.685084:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828321.685088:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828321.685094:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828321.685108:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517893632/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828321.685120:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828321.685124:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828321.685137:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828321.685149:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1328 00000100:00080000:1.0:1787828321.685156:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828321.685161:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828321.685186:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828321.685201:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517893760/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828321.685212:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828321.685220:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828321.685225:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828321.685257:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1328 00000100:00080000:1.0:1787828321.685265:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828321.685271:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828321.685281:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828321.685306:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517893888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828321.685351:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828321.685356:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828321.685358:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828321.685364:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1328 00000100:00080000:1.0:1787828321.685367:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828321.685371:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828321.685377:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828321.685409:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517894016/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828321.685423:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828321.685427:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828321.685429:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828321.685433:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1328 00000100:00080000:1.0:1787828321.685439:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828321.685445:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828321.685475:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828321.685494:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517894144/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828322.516911:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828322.708748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828323.540795:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828323.732704:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828324.564909:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828324.756831:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828325.588898:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828325.780823:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828326.612895:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828326.804733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828326.804812:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828326.804824:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828326.804830:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828326.804842:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1333 00000100:00080000:1.0:1787828326.804856:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828326.804884:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828326.804950:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517894272/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828326.804981:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828326.804985:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828326.804987:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828326.804992:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1333 00000100:00080000:1.0:1787828326.804996:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828326.805001:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828326.805029:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828326.805049:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517894400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828326.805065:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828326.805070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828326.805072:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828326.805077:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1333 00000100:00080000:1.0:1787828326.805081:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828326.805086:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828326.805091:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828326.805108:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517894528/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828326.805134:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828326.805137:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828326.805139:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828326.805142:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1333 00000100:00080000:1.0:1787828326.805145:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828326.805149:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828326.805153:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828326.805164:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517894656/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828326.805172:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828326.805176:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828326.805197:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828326.805203:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1333 00000100:00080000:1.0:1787828326.805209:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828326.805214:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828326.805222:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828326.805244:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517894784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828326.805258:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828326.805263:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828326.805266:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828326.805280:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1333 00000100:00080000:1.0:1787828326.805311:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828326.805318:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828326.805325:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828326.805345:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517894912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828326.805360:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828326.805365:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828326.805372:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828326.805380:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1333 00000100:00080000:1.0:1787828326.805384:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828326.805389:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828326.805395:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828326.805420:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517895040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828327.636850:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828327.828898:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828328.660890:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828328.852897:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828329.684830:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828329.877009:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828329.877315:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828329] req@ffff8b2e88074000 x1874672517894400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828329.877372:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517894400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828329.877390:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828329.877398:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828329.877404:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517894400:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828329.877415:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828329] req@ffff8b2e88075e00 x1874672517894528/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828329.877431:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517894528/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828329.877441:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828329.877446:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828329.877449:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517894528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828329.877458:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828329] req@ffff8b2e88074b40 x1874672517894656/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828329.877483:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517894656/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828329.877493:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828329.877498:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828329.877502:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517894656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828329.877510:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828329] req@ffff8b2e88076d00 x1874672517894272/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828329.877529:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517894272/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828329.877540:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828329.877545:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828329.877563:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517894272:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828329.877605:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828329] req@ffff8b2e88076580 x1874672517895040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828329.877625:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517895040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828329.877636:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828329.877642:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828329.877712:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517895040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828329.877738:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828329] req@ffff8b2e880770c0 x1874672517894784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828329.877758:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517894784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828329.877770:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828329.877775:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828329.877779:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517894784:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828329.877788:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828326/real 1787828329] req@ffff8b2e880761c0 x1874672517894912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828329.877806:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517894912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828351 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828329.877817:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828329.877822:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828329.877827:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517894912:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787828330.708850:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828330.900865:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828331.732946:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828331.924759:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828331.924769:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828331.924775:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828331.924789:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1338 00000100:00080000:2.0:1787828331.924805:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828331.924815:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828331.924835:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828331.924868:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517895168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828331.924904:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828331.924908:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828331.924911:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828331.924917:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1338 00000100:00080000:2.0:1787828331.924922:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828331.924928:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828331.924943:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828331.924948:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517895168:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828331.924964:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517895296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828331.924978:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828331.924982:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828331.924984:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828331.924989:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1338 00000100:00080000:2.0:1787828331.924995:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828331.924999:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828331.925006:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828331.925020:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517895424/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828331.925032:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828331.925037:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787828331.925041:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828331.925041:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828331.925049:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1338 00000100:00080000:2.0:1787828331.925055:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828331.925060:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828331.925067:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828331.925081:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517895552/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828331.925091:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828331.925095:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828331.925098:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828331.925103:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1338 00000100:00100000:0.0:1787828331.925108:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff8b2e8801c780 x1874672517895168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828331.925110:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828331.925124:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828331.925132:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828331.925137:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517895168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828331.925150:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517895680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828331.925156:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828331.925166:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828331.925168:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787828331.925174:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828331.925179:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828331.925180:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517895168:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787828331.925186:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1338 00000100:00080000:2.0:1787828331.925192:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828331.925197:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828331.925204:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828331.925218:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517895808/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828331.925230:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517895296:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828331.925232:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828331.925240:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828331.925243:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828331.925249:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1338 00000100:00080000:2.0:1787828331.925253:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828331.925259:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828331.925267:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828331.925272:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787828331.925287:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517895936/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828331.925296:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff8b2e8801ed00 x1874672517895296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828331.925316:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517895296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828331.925329:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828331.925334:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828331.925338:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517895296:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828331.925345:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517895424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828331.925390:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828331.925403:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff8b2e8801cb40 x1874672517895424/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828331.925417:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517895424/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828331.925426:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828331.925430:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828331.925434:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517895424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828331.925441:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517895552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828331.925464:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828331.925477:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff8b2e8801c3c0 x1874672517895552/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828331.925490:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517895552/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828331.925499:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828331.925502:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828331.925507:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517895552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828331.925514:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517895680:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828331.925536:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828331.925550:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff8b2e8801d680 x1874672517895680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828331.925563:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517895680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828331.925573:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828331.925576:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828331.925579:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517895680:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828331.925599:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517895808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828331.925622:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828331.925634:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff8b2e8801de00 x1874672517895808/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828331.925674:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517895808/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828331.925683:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828331.925687:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828331.925690:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517895808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828331.925697:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517895936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828331.925734:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828331.925747:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828331/real 1787828331] req@ffff8b2e8801f0c0 x1874672517895936/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828331.925761:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517895936/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828356 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828331.925770:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828331.925773:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828331.925777:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517895936:192.168.203.101@tcp:8:kworker.0 00010000:00010000:2.0:1787828332.756884:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828332.948940:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828333.781009:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828333.972755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828334.804890:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828334.996704:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828335.828931:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828336.020826:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828336.852887:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828337.044802:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828337.044819:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828337.044834:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828337.044842:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828337.044860:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1343 00000100:00080000:2.0:1787828337.044879:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828337.044894:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828337.044971:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517896064/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828337.045018:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828337.045026:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828337.045029:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828337.045038:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1343 00000100:00080000:2.0:1787828337.045048:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828337.045058:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828337.045074:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517896064:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787828337.045077:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828337.045113:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517896192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828337.045152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828337.045160:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787828337.045167:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828337.045170:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828337.045184:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1343 00000100:00080000:2.0:1787828337.045195:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828337.045204:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828337.045216:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828337.045228:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828337/real 1787828337] req@ffff8b2e8801f0c0 x1874672517896064/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828337.045247:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517896320/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828337.045261:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517896064/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828337.045285:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828337.045298:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828337.045300:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828337.045304:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828337.045316:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787828337.045320:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1343 00000100:00080000:2.0:1787828337.045333:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828337.045334:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517896064:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787828337.045350:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828337.045365:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828337.045387:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517896192:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787828337.045395:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517896448/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828337.045423:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828337.045431:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828337.045436:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828337.045451:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1343 00000800:00000100:0.0:1787828337.045454:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828337.045462:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828337.045474:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828337.045485:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828337.045494:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828337/real 1787828337] req@ffff8b2e8801de00 x1874672517896192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828337.045517:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517896576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828337.045530:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517896192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828337.045550:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828337.045568:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828337.045571:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828337.045573:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828337.045588:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787828337.045594:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1343 00000100:00100000:0.0:1787828337.045603:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517896192:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828337.045610:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828337.045619:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517896320:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787828337.045630:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828337.045643:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828337.045725:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828337.045750:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828337/real 1787828337] req@ffff8b2e8801d680 x1874672517896320/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828337.045762:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517896704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828337.045779:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517896320/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828337.045801:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828337.045804:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828337.045817:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787828337.045821:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828337.045825:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517896320:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787828337.045831:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828337.045844:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1343 00000100:00080000:2.0:1787828337.045854:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828337.045856:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517896448:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787828337.045872:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828337.045886:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828337.045896:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828337.045916:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828337/real 1787828337] req@ffff8b2e8801c3c0 x1874672517896448/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828337.045926:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517896832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828337.045942:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517896448/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828337.045970:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828337.045976:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828337.045982:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517896448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828337.045991:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517896576:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828337.046013:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828337.046025:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828337/real 1787828337] req@ffff8b2e8801cb40 x1874672517896576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828337.046039:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517896576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828337.046050:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828337.046053:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828337.046057:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517896576:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828337.046063:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517896704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828337.046080:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828337.046091:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828337/real 1787828337] req@ffff8b2e8801ed00 x1874672517896704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828337.046105:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517896704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828337.046114:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828337.046118:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828337.046122:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517896704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828337.046139:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517896832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828337.046158:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828337.046169:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828337/real 1787828337] req@ffff8b2e8801c780 x1874672517896832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828337.046183:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517896832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828362 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828337.046193:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828337.046197:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828337.046200:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517896832:192.168.203.101@tcp:8:kworker.0 00010000:00010000:2.0:1787828337.876844:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828338.068756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828338.900812:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828339.092801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828339.924832:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828340.116808:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828340.948891:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828341.140780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828341.972894:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828342.164738:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828342.164741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828342.164751:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828342.164755:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828342.164766:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1348 00000100:00080000:2.0:1787828342.164779:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828342.164788:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828342.164834:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517896960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828342.164870:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828342.164876:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828342.164879:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828342.164886:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1348 00000100:00080000:2.0:1787828342.164893:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828342.164899:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828342.164910:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517896960:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787828342.164911:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828342.164933:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517897088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828342.164945:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828342.164949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828342.164951:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828342.164955:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1348 00000100:00080000:2.0:1787828342.164959:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828342.164962:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828342.164967:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828342.164979:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517897216/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828342.164985:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828342.164997:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828342.165000:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828342.165002:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828342.165007:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1348 00000100:00080000:2.0:1787828342.165011:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828342.165014:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828342.165017:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828342.165027:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517897344/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828342.165037:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828342/real 1787828342] req@ffff8b2e8801da40 x1874672517896960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828342.165044:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828342.165058:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828342.165063:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828342.165069:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672517896960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828342.165075:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1348 00000100:00080000:2.0:1787828342.165087:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828342.165095:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828342.165096:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828342.165109:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:2.0:1787828342.165109:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828342.165122:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517896960:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828342.165129:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517897472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828342.165141:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828342.165144:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828342.165146:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828342.165153:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1348 00000100:00100000:0.0:1787828342.165160:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517897088:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828342.165161:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828342.165173:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828342.165181:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828342.165199:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517897600/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828342.165205:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828342.165223:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828342.165227:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828342.165229:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828342.165234:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828342/real 1787828342] req@ffff8b2e8801fc00 x1874672517897088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828342.165236:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1348 00000100:00080000:2.0:1787828342.165243:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828342.165251:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828342.165262:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828342.165271:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517897088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828342.165283:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517897728/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828342.165284:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828342.165294:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828342.165303:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517897088:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828342.165317:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517897216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828342.165343:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828342.165354:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828342/real 1787828342] req@ffff8b2e8801cf00 x1874672517897216/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828342.165364:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672517897216/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828342.165372:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828342.165374:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828342.165377:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517897216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828342.165382:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517897344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828342.165395:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828342.165403:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828342/real 1787828342] req@ffff8b2e8801c000 x1874672517897344/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828342.165413:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672517897344/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828342.165420:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828342.165423:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828342.165425:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517897344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828342.165430:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517897472:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828342.165442:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828342.165450:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828342/real 1787828342] req@ffff8b2e8801e580 x1874672517897472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828342.165460:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517897472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828342.165467:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828342.165470:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828342.165472:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517897472:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828342.165488:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517897600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828342.165507:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828342.165522:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828342/real 1787828342] req@ffff8b2e8801e940 x1874672517897600/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828342.165538:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517897600/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828342.165551:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828342.165557:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828342.165561:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517897600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828342.165570:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517897728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828342.165590:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828342.165600:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828342/real 1787828342] req@ffff8b2e8801f480 x1874672517897728/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828342.165611:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517897728/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828367 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828342.165618:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828342.165621:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828342.165623:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517897728:192.168.203.101@tcp:8:kworker.0 00010000:00010000:1.0:1787828342.996777:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828343.188771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828344.020802:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828344.212780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828345.044905:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00010000:00010000:1.0:1787828346.068937:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828346.260887:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828347.092834:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828347.284744:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828347.284751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828347.284756:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828347.284760:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828347.284769:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1353 00000100:00080000:0.0:1787828347.284779:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828347.284786:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828347.284819:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517897856/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828347.284843:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828347.284846:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828347.284847:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828347.284850:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1353 00000100:00080000:0.0:1787828347.284853:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828347.284856:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828347.284864:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828347.284873:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517897984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828347.284890:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828347.284893:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828347.284894:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828347.284896:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1353 00000100:00080000:0.0:1787828347.284900:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828347.284902:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828347.284905:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828347.284914:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517898112/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828347.284920:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828347.284922:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828347.284923:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828347.284925:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1353 00000100:00080000:0.0:1787828347.284934:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828347.284937:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828347.284940:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828347.284952:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517898240/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828347.284959:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828347.284962:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828347.284964:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828347.284966:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1353 00000100:00080000:0.0:1787828347.284971:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828347.284973:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828347.284975:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828347.284982:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517898368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828347.284989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828347.284991:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828347.284993:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828347.285004:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1353 00000100:00080000:0.0:1787828347.285010:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828347.285016:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828347.285020:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828347.285034:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517898496/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828347.285086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828347.285094:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828347.285100:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828347.285110:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1353 00000100:00080000:0.0:1787828347.285117:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828347.285120:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828347.285125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828347.285135:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517898624/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828348.116903:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828348.308785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828349.140897:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828349.332835:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828350.164863:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828350.356816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828351.188904:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828351.380782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828352.212862:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828352.404795:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828352.404808:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787828352.404809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828352.404815:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828352.404835:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1358 00000100:00080000:0.0:1787828352.404852:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828352.404863:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828352.404916:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517898752/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828352.404968:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828352.404973:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828352.404976:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828352.404981:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1358 00000100:00080000:0.0:1787828352.404984:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828352.404988:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828352.405000:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828352.405017:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517898880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828352.405031:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828352.405036:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828352.405037:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828352.405041:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1358 00000100:00080000:0.0:1787828352.405044:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828352.405046:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828352.405051:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828352.405061:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517899008/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828352.405069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828352.405072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828352.405073:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828352.405076:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1358 00000100:00080000:0.0:1787828352.405079:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828352.405082:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828352.405094:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828352.405104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517899136/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828352.405120:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828352.405123:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828352.405125:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828352.405129:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1358 00000100:00080000:0.0:1787828352.405133:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828352.405136:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828352.405140:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828352.405149:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517899264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828352.405156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828352.405160:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828352.405161:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828352.405165:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1358 00000100:00080000:0.0:1787828352.405168:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828352.405182:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828352.405188:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828352.405202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517899392/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828352.405228:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828352.405236:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828352.405239:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828352.405247:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1358 00000100:00080000:0.0:1787828352.405254:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828352.405258:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828352.405261:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828352.405270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517899520/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828353.236965:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828353.428909:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828354.260861:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828354.452779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828355.284850:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828355.476805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828356.308847:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828356.500777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828357.332897:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828357.524810:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.524831:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828357.524840:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.524860:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1363 00000100:00100000:2.0:1787828357.524873:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828357.524880:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.524896:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828357.524911:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828357.524964:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517899648/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828357.525009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.525014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828357.525017:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.525033:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1363 00000100:00080000:0.0:1787828357.525038:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.525044:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828357.525058:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828357.525086:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517899776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828357.525103:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.525108:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828357.525112:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.525123:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1363 00000100:00080000:0.0:1787828357.525129:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.525140:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828357.525151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828357.525167:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517899904/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828357.525180:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.525184:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828357.525186:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.525190:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1363 00000100:00080000:0.0:1787828357.525194:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.525198:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828357.525203:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828357.525215:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517900032/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828357.525227:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.525231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828357.525234:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.525241:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1363 00000100:00080000:0.0:1787828357.525246:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.525251:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828357.525258:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828357.525280:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517900160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828357.525312:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.525320:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828357.525325:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.525336:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1363 00000100:00080000:0.0:1787828357.525344:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.525353:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828357.525364:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828357.525388:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517900288/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828357.525428:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828357.525441:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828357.525445:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828357.525451:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1363 00000100:00080000:0.0:1787828357.525458:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828357.525468:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828357.525474:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828357.525495:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517900416/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828358.356918:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828358.548809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828359.380879:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828359.572775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828360.404866:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828360.596750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828361.428770:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828361.620725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828362.452871:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828362.644763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828362.644789:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828362.644803:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828362.644810:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828362.644825:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1369 00000100:00080000:1.0:1787828362.644840:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828362.644850:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828362.644936:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517900544/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828362.645009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828362.645016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828362.645019:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828362.645027:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1369 00000100:00080000:1.0:1787828362.645034:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828362.645043:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828362.645059:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828362.645085:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517900672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828362.645100:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828362.645105:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828362.645107:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828362.645111:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1369 00000100:00080000:1.0:1787828362.645116:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828362.645120:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828362.645126:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828362.645140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517900800/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828362.645157:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828362.645161:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828362.645164:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828362.645168:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1369 00000100:00080000:1.0:1787828362.645172:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828362.645175:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828362.645178:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828362.645186:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517900928/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828362.645195:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828362.645199:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828362.645202:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828362.645205:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1369 00000100:00080000:1.0:1787828362.645210:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828362.645214:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828362.645219:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828362.645234:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517901056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828362.645263:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828362.645271:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828362.645274:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828362.645283:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1369 00000100:00080000:1.0:1787828362.645299:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828362.645308:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828362.645316:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828362.645327:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517901184/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828362.645337:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828362.645343:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828362.645349:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828362.645355:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1369 00000100:00080000:1.0:1787828362.645358:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828362.645362:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828362.645368:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828362.645394:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517901312/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828363.476837:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828363.476927:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828363.668751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828364.500792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828364.500911:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828364.692715:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828365.524771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828365.524930:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828365.716750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828366.548731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828366.548803:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828366.740717:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828367.572743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828367.572794:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828367.764771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828367.764861:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828367.765066:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517901440:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828367.765172:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828367.765241:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828367] req@ffff8b2e88039680 x1874672517901440/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828367.765282:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517901440/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828367.765322:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828367.765335:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828367.765349:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517901440:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828367.765398:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517901568:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828367.765446:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828367.765497:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828367] req@ffff8b2e88039e00 x1874672517901568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828367.765526:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517901568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828367.765569:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828367.765575:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828367.765581:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517901568:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828367.765593:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517901696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828367.765622:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828367.765636:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828367] req@ffff8b2e8803a1c0 x1874672517901696/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828367.765686:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517901696/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828367.765706:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828367.765710:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828367.765714:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517901696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828367.765722:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517901824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828367.765746:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828367.765759:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828367] req@ffff8b2e8803b480 x1874672517901824/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828367.765773:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517901824/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828367.765784:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828367.765788:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828367.765791:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517901824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828367.765798:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517901952:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828367.765816:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828367.765827:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828367] req@ffff8b2e880392c0 x1874672517901952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828367.765840:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517901952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828367.765850:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828367.765853:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828367.765858:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517901952:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828367.765880:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517902080:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828367.765899:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828367.765911:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828367] req@ffff8b2e880383c0 x1874672517902080/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828367.765925:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517902080/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828367.765935:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828367.765938:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828367.765942:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517902080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828367.765948:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517902208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828367.765966:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828367.765977:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828367/real 1787828367] req@ffff8b2e8803a580 x1874672517902208/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828367.765991:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517902208/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828392 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828367.766001:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828367.766004:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828367.766008:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517902208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828368.596866:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828368.596895:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828368.788754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828368.788754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828369.620790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828369.620850:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828369.812803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828369.812816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828370.644822:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828370.644882:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828370.836723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828370.836726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828371.668751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828371.668813:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828371.860770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828371.860818:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828372.692733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828372.692804:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828372.884724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828372.884774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828372.884926:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517902336:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828372.885021:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828372.885082:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff8b2e88039a40 x1874672517902336/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828372.885114:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517902336/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828372.885127:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828372.885133:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828372.885139:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517902336:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828372.885171:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517902464:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828372.885200:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828372.885212:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff8b2e8803b0c0 x1874672517902464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828372.885226:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517902464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828372.885237:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828372.885240:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828372.885244:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517902464:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828372.885252:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517902592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828372.885275:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828372.885311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff8b2e88038f00 x1874672517902592/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828372.885320:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517902592/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828372.885332:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828372.885334:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828372.885337:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517902592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828372.885342:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517902720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828372.885372:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828372.885380:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff8b2e8803b840 x1874672517902720/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828372.885388:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517902720/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828372.885395:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828372.885397:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828372.885399:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517902720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828372.885404:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517902848:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828372.885416:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828372.885437:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff8b2e8803a580 x1874672517902848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828372.885464:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517902848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828372.885470:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828372.885473:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828372.885482:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517902848:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828372.885489:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517902976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828372.885513:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828372.885526:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff8b2e880383c0 x1874672517902976/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828372.885538:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517902976/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828372.885544:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828372.885546:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828372.885548:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517902976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828372.885553:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517903104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828372.885569:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828372.885576:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828372/real 1787828372] req@ffff8b2e880392c0 x1874672517903104/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828372.885584:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517903104/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828397 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828372.885590:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828372.885591:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828372.885593:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517903104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828373.716763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828373.716887:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828373.908790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828373.908835:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828374.740810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828374.932835:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828374.932841:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828375.764777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828375.764994:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828375.956751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828376.788762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828376.788870:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828376.980832:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828377.812765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:0.0:1787828378.004744:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828378.004757:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787828378.004764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828378.004766:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828378.004787:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1384 00000100:00080000:0.0:1787828378.004800:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828378.004809:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828378.004851:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517903232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828378.004876:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828378.004879:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828378.004880:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828378.004883:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1384 00000100:00080000:0.0:1787828378.004886:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828378.004888:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828378.004898:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828378.004910:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517903360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828378.004918:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828378.004923:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828378.004925:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828378.004933:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1384 00000100:00080000:0.0:1787828378.004943:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828378.004946:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828378.004950:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828378.004962:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517903488/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828378.004974:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828378.004976:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828378.004978:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828378.004980:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1384 00000100:00080000:0.0:1787828378.004983:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828378.004985:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828378.004988:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828378.004996:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517903616/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828378.005002:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828378.005005:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828378.005006:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828378.005009:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1384 00000100:00080000:0.0:1787828378.005012:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828378.005016:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828378.005020:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828378.005029:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517903744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828378.005036:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828378.005038:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828378.005039:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828378.005042:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1384 00000100:00080000:0.0:1787828378.005047:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828378.005050:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828378.005053:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828378.005060:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517903872/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828378.005065:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828378.005068:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828378.005070:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828378.005073:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1384 00000100:00080000:0.0:1787828378.005077:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828378.005082:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828378.005088:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828378.005100:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517904000/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828378.836778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828379.028769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828379.860791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828379.860864:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828380.052716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828380.884884:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828381.076764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828381.908815:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828382.100808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828382.932782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828382.932783:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828383.124807:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828383.124807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828383.124823:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828383.124829:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828383.124841:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1389 00000100:00080000:1.0:1787828383.124856:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828383.124866:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828383.124929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517904128/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828383.124965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828383.124969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828383.124971:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828383.124976:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1389 00000100:00080000:1.0:1787828383.124979:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828383.124982:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828383.124994:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828383.125007:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517904256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828383.125019:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828383.125026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828383.125029:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828383.125040:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1389 00000100:00080000:1.0:1787828383.125044:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828383.125047:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828383.125053:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828383.125078:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517904384/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828383.125088:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828383.125092:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828383.125095:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828383.125099:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1389 00000100:00080000:1.0:1787828383.125103:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828383.125107:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828383.125112:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828383.125123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517904512/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828383.125139:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828383.125146:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828383.125147:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828383.125152:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1389 00000100:00080000:1.0:1787828383.125155:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828383.125159:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828383.125163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828383.125173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517904640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828383.125183:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828383.125186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828383.125189:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828383.125192:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1389 00000100:00080000:1.0:1787828383.125195:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828383.125199:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828383.125203:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828383.125216:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517904768/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828383.125233:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828383.125243:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828383.125247:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828383.125265:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1389 00000100:00080000:1.0:1787828383.125270:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828383.125273:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828383.125281:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828383.125298:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517904896/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828383.956729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828384.148807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828384.980795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828385.172847:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828386.004801:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828386.004857:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828386.196720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828387.028731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828387.220785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828388.052771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828388.244773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828388.244848:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828388.244871:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828388.244877:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828388.244892:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1394 00000100:00080000:1.0:1787828388.244908:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828388.244917:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828388.244979:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517905024/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828388.245010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828388.245015:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828388.245018:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828388.245024:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1394 00000100:00080000:1.0:1787828388.245030:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828388.245036:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828388.245054:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828388.245088:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517905152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828388.245107:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828388.245112:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828388.245114:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828388.245118:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1394 00000100:00080000:1.0:1787828388.245126:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828388.245132:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828388.245138:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828388.245151:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517905280/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828388.245170:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828388.245182:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828388.245191:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828388.245197:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1394 00000100:00080000:1.0:1787828388.245201:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828388.245206:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828388.245211:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828388.245221:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517905408/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828388.245237:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828388.245241:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828388.245243:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828388.245247:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1394 00000100:00080000:1.0:1787828388.245251:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828388.245255:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828388.245259:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828388.245275:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517905536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828388.245351:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828388.245361:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828388.245365:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828388.245373:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1394 00000100:00080000:1.0:1787828388.245377:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828388.245388:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828388.245398:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828388.245421:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517905664/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828388.245433:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828388.245444:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828388.245448:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828388.245457:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1394 00000100:00080000:1.0:1787828388.245466:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828388.245475:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828388.245480:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828388.245502:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517905792/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828389.076824:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828389.076916:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828389.268834:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828390.100768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828390.292761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828391.124826:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828391.316808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828392.148777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828392.148878:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828392.340741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828393.172760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828393.364812:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828393.364822:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.364838:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828393.364846:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.364865:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1399 00000100:00080000:1.0:1787828393.364885:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.364900:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828393.364955:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517905920/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828393.364997:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.365018:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828393.365023:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.365033:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1399 00000100:00080000:1.0:1787828393.365043:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.365052:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828393.365076:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828393.365111:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517906048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828393.365138:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.365144:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828393.365147:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.365152:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1399 00000100:00080000:1.0:1787828393.365158:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.365164:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828393.365171:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828393.365189:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517906176/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828393.365202:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.365207:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828393.365211:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.365218:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1399 00000100:00080000:1.0:1787828393.365223:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.365229:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828393.365237:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828393.365261:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517906304/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828393.365279:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.365315:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828393.365319:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.365326:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1399 00000100:00080000:1.0:1787828393.365334:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.365347:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828393.365361:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828393.365385:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517906432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828393.365405:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.365422:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828393.365425:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.365431:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1399 00000100:00080000:1.0:1787828393.365435:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.365439:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828393.365447:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828393.365467:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517906560/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828393.365481:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828393.365486:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828393.365490:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828393.365496:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1399 00000100:00080000:1.0:1787828393.365506:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828393.365517:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828393.365528:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828393.365545:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517906688/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828394.196772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828394.388774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828395.220760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828395.220828:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828395.412757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828396.244723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828396.436804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:1.0:1787828396.437247:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787828396.437337:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828396.437354:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787828396.437391:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787828396.437401:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787828396.437426:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787828396.437441:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787828396.437455:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787828396.437473:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:2.0:1787828397.268742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828397.460769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828398.292742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828398.292868:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828398.484738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828398.484843:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828398.484849:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828398.484853:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828398.484863:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1404 00000100:00080000:0.0:1787828398.484873:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828398.484880:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828398.484914:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517906816/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828398.484955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828398.484960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828398.484962:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828398.484965:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1404 00000100:00080000:0.0:1787828398.484968:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828398.484971:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828398.484980:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828398.484991:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517906944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828398.485002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828398.485006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828398.485011:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828398.485020:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1404 00000100:00080000:0.0:1787828398.485024:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828398.485028:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828398.485047:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828398.485067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517907072/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828398.485081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828398.485084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828398.485087:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828398.485095:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1404 00000100:00080000:0.0:1787828398.485102:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828398.485110:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828398.485118:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828398.485134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517907200/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828398.485147:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828398.485152:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828398.485155:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828398.485164:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1404 00000100:00080000:0.0:1787828398.485173:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828398.485183:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828398.485193:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828398.485211:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517907328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828398.485236:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828398.485244:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828398.485263:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828398.485278:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1404 00000100:00080000:0.0:1787828398.485287:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828398.485296:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828398.485303:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828398.485323:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517907456/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828398.485338:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828398.485343:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828398.485347:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828398.485352:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1404 00000100:00080000:0.0:1787828398.485358:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828398.485367:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828398.485376:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828398.485402:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517907584/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828399.316783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828399.508732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828400.340766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828400.532737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828401.364762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828401.364807:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828401.556769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828402.388761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828402.580787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828403.412771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828403.604802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828403.604814:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828403.604823:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828403.604828:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828403.604841:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1410 00000100:00080000:0.0:1787828403.604856:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828403.604866:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828403.604912:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517907712/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828403.604950:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828403.604955:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828403.604958:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828403.604963:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1410 00000100:00080000:0.0:1787828403.604969:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828403.604974:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828403.604986:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828403.605006:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517907840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828403.605034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828403.605041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828403.605043:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828403.605052:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1410 00000100:00080000:0.0:1787828403.605058:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828403.605064:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828403.605071:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828403.605091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517907968/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828403.605105:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828403.605111:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828403.605114:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828403.605128:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1410 00000100:00080000:0.0:1787828403.605137:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828403.605143:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828403.605152:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828403.605172:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517908096/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828403.605199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828403.605206:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828403.605229:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828403.605250:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1410 00000100:00080000:0.0:1787828403.605270:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828403.605288:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828403.605306:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828403.605337:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517908224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828403.605362:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828403.605367:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828403.605376:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828403.605390:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1410 00000100:00080000:0.0:1787828403.605409:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828403.605423:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828403.605448:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828403.605478:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517908352/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828403.605497:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828403.605508:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828403.605514:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828403.605524:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1410 00000100:00080000:0.0:1787828403.605544:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828403.605558:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828403.605581:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828403.605612:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517908480/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828404.436791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828404.437006:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828404.628778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828405.460755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828405.652798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828406.484768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828406.676734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828407.508756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828407.508810:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828407.700741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828408.532737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:0.0:1787828408.724859:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828408.724862:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828408.724878:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828408.724884:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828408.724897:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1415 00000100:00080000:0.0:1787828408.724914:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828408.724927:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828408.724996:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517908608/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828408.725032:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828408.725038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828408.725040:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828408.725046:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1415 00000100:00080000:0.0:1787828408.725053:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828408.725060:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828408.725073:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828408.725118:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517908736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828408.725143:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828408.725150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828408.725155:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828408.725200:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1415 00000100:00080000:0.0:1787828408.725208:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828408.725214:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828408.725224:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828408.725249:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517908864/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828408.725267:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828408.725272:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828408.725275:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828408.725282:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1415 00000100:00080000:0.0:1787828408.725289:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828408.725302:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828408.725315:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828408.725340:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517908992/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828408.725398:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828408.725404:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828408.725412:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828408.725424:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1415 00000100:00080000:0.0:1787828408.725440:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828408.725460:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828408.725472:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828408.725498:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517909120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828408.725594:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828408.725611:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828408.725617:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828408.725629:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1415 00000100:00080000:0.0:1787828408.725640:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828408.725697:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828408.725703:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828408.725718:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517909248/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828408.725733:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828408.725738:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828408.725740:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828408.725746:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1415 00000100:00080000:0.0:1787828408.725751:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828408.725755:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828408.725762:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828408.725777:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517909376/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828409.556760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828409.748763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828410.580741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828410.580827:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828410.772780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828411.604747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828411.796721:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828412.628757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828412.820846:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828413.652761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828413.652852:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828413.844759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828413.844965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828413.844972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828413.844977:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828413.844990:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1420 00000100:00080000:2.0:1787828413.845005:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828413.845016:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828413.845074:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517909504/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828413.845108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828413.845112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828413.845115:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828413.845119:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1420 00000100:00080000:2.0:1787828413.845124:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828413.845129:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828413.845160:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828413.845203:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517909632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828413.845229:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828413.845235:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828413.845238:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828413.845242:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1420 00000100:00080000:2.0:1787828413.845248:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828413.845254:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828413.845262:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828413.845278:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517909760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828413.845333:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828413.845339:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828413.845344:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828413.845352:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1420 00000100:00080000:2.0:1787828413.845361:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828413.845368:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828413.845379:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828413.845411:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517909888/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828413.845469:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828413.845481:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828413.845490:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828413.845504:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1420 00000100:00080000:2.0:1787828413.845514:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828413.845522:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828413.845536:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828413.845566:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517910016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828413.845589:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828413.845606:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828413.845612:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828413.845627:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1420 00000100:00080000:2.0:1787828413.845637:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828413.845650:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828413.845659:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828413.845708:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517910144/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828413.845721:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828413.845725:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828413.845727:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828413.845731:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1420 00000100:00080000:2.0:1787828413.845736:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828413.845746:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828413.845757:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828413.845776:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517910272/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:0.0:1787828414.676820:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828414.868821:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828415.700786:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828415.892789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828416.724881:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828416.916832:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828417.940769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828418.772866:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828418.964795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828418.964826:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828418.964837:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828418.964844:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828418.964860:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1425 00000100:00080000:1.0:1787828418.964876:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828418.964886:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828418.964939:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880752c0 x1874672517910400/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828418.964993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828418.964998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828418.965001:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828418.965006:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1425 00000100:00080000:1.0:1787828418.965010:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828418.965016:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828418.965040:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828418.965057:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517910528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828418.965078:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828418.965085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828418.965087:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828418.965092:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1425 00000100:00080000:1.0:1787828418.965096:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828418.965101:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828418.965144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828418.965166:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517910656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828418.965181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828418.965185:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828418.965188:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828418.965193:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1425 00000100:00080000:1.0:1787828418.965204:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828418.965227:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828418.965238:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828418.965267:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517910784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828418.965290:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828418.965309:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828418.965319:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828418.965332:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1425 00000100:00080000:1.0:1787828418.965346:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828418.965363:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828418.965377:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828418.965426:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517910912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828418.965455:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828418.965465:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828418.965470:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828418.965482:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1425 00000100:00080000:1.0:1787828418.965491:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828418.965507:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828418.965522:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828418.965563:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517911040/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828418.965691:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828418.965712:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828418.965729:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828418.965744:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1425 00000100:00080000:1.0:1787828418.965754:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828418.965762:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828418.965779:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828418.965816:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517911168/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828419.796817:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828419.988811:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828420.820917:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828421.012787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828421.844850:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828422.036772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828422.868787:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828423.060740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828423.892874:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828424.084739:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828424.084750:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828424.084755:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828424.084762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828424.084765:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1430 00000100:00080000:1.0:1787828424.084781:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828424.084793:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828424.084831:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517911296/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828424.084866:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828424.084869:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828424.084871:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828424.084874:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1430 00000100:00080000:1.0:1787828424.084877:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828424.084880:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828424.084889:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828424.084899:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517911424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828424.084909:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828424.084911:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828424.084912:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828424.084915:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1430 00000100:00080000:1.0:1787828424.084918:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828424.084925:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828424.084940:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828424.084955:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517911552/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828424.084962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828424.084966:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828424.084968:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828424.084971:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1430 00000100:00080000:1.0:1787828424.084973:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828424.084975:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828424.084979:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828424.084987:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517911680/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828424.084994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828424.084996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828424.084997:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828424.085000:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1430 00000100:00080000:1.0:1787828424.085003:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828424.085015:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828424.085018:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828424.085025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517911808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828424.085030:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828424.085032:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828424.085034:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828424.085036:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1430 00000100:00080000:1.0:1787828424.085041:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828424.085044:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828424.085046:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828424.085054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517911936/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828424.085084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828424.085090:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828424.085092:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828424.085097:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1430 00000100:00080000:1.0:1787828424.085103:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828424.085107:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828424.085130:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828424.085140:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517912064/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828424.916783:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828425.108770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828425.940852:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828426.132877:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828426.964995:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828427.156843:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828427.988786:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828428.180844:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828429.012909:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828429.204835:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828429.204837:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828429.204861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828429.204868:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828429.204881:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1435 00000100:00080000:2.0:1787828429.204897:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828429.204908:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828429.204956:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517912192/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828429.204994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828429.204998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828429.205001:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828429.205006:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1435 00000100:00080000:2.0:1787828429.205010:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828429.205016:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828429.205028:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828429.205102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517912320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828429.205120:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828429.205125:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828429.205129:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828429.205134:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1435 00000100:00080000:2.0:1787828429.205177:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828429.205183:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828429.205190:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828429.205214:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517912448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828429.205228:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828429.205232:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828429.205235:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828429.205240:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1435 00000100:00080000:2.0:1787828429.205246:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828429.205251:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828429.205256:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828429.205274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517912576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828429.205308:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828429.205320:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828429.205325:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828429.205342:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1435 00000100:00080000:2.0:1787828429.205361:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828429.205374:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828429.205384:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828429.205398:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517912704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828429.205413:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828429.205425:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828429.205427:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828429.205436:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1435 00000100:00080000:2.0:1787828429.205447:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828429.205464:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828429.205477:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828429.205527:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517912832/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828429.205597:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828429.205615:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828429.205623:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828429.205637:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1435 00000100:00080000:2.0:1787828429.205682:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828429.205691:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828429.205699:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828429.205717:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517912960/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828430.036763:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828430.228797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828431.060829:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828431.252801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828432.084856:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828432.276731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828433.108846:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828433.300741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828434.132882:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828434.324766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828434.324812:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.324828:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828434.324836:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.324852:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1440 00000100:00080000:0.0:1787828434.324880:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.324893:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828434.324947:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517913088/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828434.324985:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.324989:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828434.324992:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.325019:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1440 00000100:00080000:0.0:1787828434.325034:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.325040:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828434.325053:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828434.325077:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517913216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828434.325121:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.325126:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828434.325128:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.325134:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1440 00000100:00080000:0.0:1787828434.325138:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.325143:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828434.325149:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828434.325164:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517913344/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828434.325190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.325195:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828434.325199:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.325205:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1440 00000100:00080000:0.0:1787828434.325214:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.325218:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828434.325223:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828434.325239:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517913472/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828434.325264:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.325277:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828434.325282:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.325297:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1440 00000100:00080000:0.0:1787828434.325303:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.325309:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828434.325319:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828434.325357:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517913600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828434.325390:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.325403:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828434.325407:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.325416:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1440 00000100:00080000:0.0:1787828434.325422:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.325433:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828434.325440:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828434.325457:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517913728/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828434.325482:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828434.325486:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828434.325492:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828434.325505:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1440 00000100:00080000:0.0:1787828434.325514:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828434.325524:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828434.325533:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828434.325555:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517913856/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787828435.348817:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828436.180886:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828436.372768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828437.204901:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828437.396848:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828438.228878:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828438.420849:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828439.252881:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828439.444869:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828439.444880:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1787828439.444882:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828439.444888:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828439.444903:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1445 00000100:00080000:0.0:1787828439.444922:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828439.444935:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828439.445027:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517913984/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828439.445071:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828439.445076:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828439.445079:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828439.445084:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1445 00000100:00080000:0.0:1787828439.445088:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828439.445093:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828439.445107:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828439.445122:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517914112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828439.445136:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828439.445145:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828439.445151:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828439.445157:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1445 00000100:00080000:0.0:1787828439.445160:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828439.445164:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828439.445169:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828439.445190:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517914240/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828439.445201:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828439.445204:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828439.445206:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828439.445210:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1445 00000100:00080000:0.0:1787828439.445215:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828439.445220:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828439.445224:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828439.445237:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517914368/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828439.445246:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828439.445250:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828439.445252:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828439.445264:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1445 00000100:00080000:0.0:1787828439.445271:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828439.445276:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828439.445281:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828439.445293:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517914496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828439.445303:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828439.445307:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828439.445309:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828439.445313:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1445 00000100:00080000:0.0:1787828439.445316:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828439.445321:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828439.445326:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828439.445343:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517914624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828439.445369:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828439.445384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828439.445388:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828439.445393:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1445 00000100:00080000:0.0:1787828439.445395:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828439.445400:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828439.445411:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828439.445426:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517914752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828440.276879:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828440.468754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828441.300920:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828441.492778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828442.324911:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828442.516747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828443.348910:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828443.540723:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828444.372825:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828444.564748:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828444.564786:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828444.564790:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828444.564801:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1450 00000100:00100000:1.0:1787828444.564806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828444.564813:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828444.564839:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828444.564895:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517914880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828444.564922:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828444.564940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828444.564959:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828444.564975:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1450 00000100:00080000:0.0:1787828444.564979:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828444.564982:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828444.564994:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828444.565009:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517915008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828444.565019:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828444.565022:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828444.565024:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828444.565037:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1450 00000100:00080000:0.0:1787828444.565039:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828444.565042:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828444.565046:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828444.565055:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517915136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828444.565062:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828444.565065:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828444.565066:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828444.565069:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1450 00000100:00080000:0.0:1787828444.565071:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828444.565073:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828444.565076:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828444.565085:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517915264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828444.565104:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828444.565106:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828444.565108:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828444.565112:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1450 00000100:00080000:0.0:1787828444.565124:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828444.565129:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828444.565135:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828444.565151:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517915392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828444.565167:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828444.565173:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828444.565178:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828444.565186:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1450 00000100:00080000:0.0:1787828444.565192:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828444.565198:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828444.565203:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828444.565213:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517915520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828444.565231:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828444.565234:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828444.565236:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828444.565239:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1450 00000100:00080000:0.0:1787828444.565243:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828444.565248:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828444.565253:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828444.565263:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517915648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828445.396870:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828445.588733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828446.420791:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828446.612732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828447.444811:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828447.636751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828448.468820:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828448.660706:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828449.492813:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828449.684767:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828449.684779:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828449.684783:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828449.684788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828449.684794:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1456 00000100:00080000:2.0:1787828449.684805:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828449.684813:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828449.684855:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517915776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828449.684878:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828449.684881:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828449.684882:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828449.684885:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1456 00000100:00080000:2.0:1787828449.684902:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828449.684906:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828449.684915:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828449.684928:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517915904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828449.684937:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828449.684940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828449.684941:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828449.684944:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1456 00000100:00080000:2.0:1787828449.684947:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828449.684949:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828449.684953:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828449.684961:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517916032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828449.684967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828449.684970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828449.684971:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828449.684978:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1456 00000100:00080000:2.0:1787828449.684982:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828449.684985:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828449.684988:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828449.684996:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517916160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828449.685006:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828449.685008:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828449.685010:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828449.685013:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1456 00000100:00080000:2.0:1787828449.685016:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828449.685023:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828449.685029:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828449.685041:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517916288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828449.685049:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828449.685055:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828449.685058:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828449.685064:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1456 00000100:00080000:2.0:1787828449.685068:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828449.685071:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828449.685075:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828449.685085:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517916416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828449.685098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828449.685101:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828449.685102:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828449.685105:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1456 00000100:00080000:2.0:1787828449.685108:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828449.685111:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828449.685113:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828449.685122:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517916544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828450.516753:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828450.708743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828451.540954:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828451.732799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828452.564826:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828452.756738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828453.588842:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828453.780749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828454.612975:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828454.804769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828454.804775:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828454.804787:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828454.804793:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828454.804806:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1461 00000100:00080000:2.0:1787828454.804818:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828454.804828:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828454.804904:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517916672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828454.804947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828454.804954:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828454.804957:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828454.804965:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1461 00000100:00080000:2.0:1787828454.804974:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828454.804980:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828454.804994:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828454.805017:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517916800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828454.805033:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828454.805038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828454.805041:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828454.805046:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1461 00000100:00080000:2.0:1787828454.805062:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828454.805066:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828454.805072:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828454.805093:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517916928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828454.805106:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828454.805110:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828454.805112:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828454.805116:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1461 00000100:00080000:2.0:1787828454.805121:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828454.805126:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828454.805131:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828454.805148:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517917056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828454.805161:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828454.805168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828454.805174:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828454.805187:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1461 00000100:00080000:2.0:1787828454.805196:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828454.805202:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828454.805214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828454.805242:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517917184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828454.805264:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828454.805273:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828454.805283:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828454.805292:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1461 00000100:00080000:2.0:1787828454.805297:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828454.805303:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828454.805310:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828454.805330:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517917312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828454.805347:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828454.805357:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828454.805362:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828454.805371:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1461 00000100:00080000:2.0:1787828454.805375:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828454.805378:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828454.805385:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828454.805405:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517917440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828455.636810:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828455.828886:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828456.660859:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828456.852763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828457.684951:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828457.876782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828458.708882:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828458.900813:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828459.732788:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828459.924765:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828459.924780:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787828459.924791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828459.924792:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828459.924809:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1466 00000100:00080000:2.0:1787828459.924847:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828459.924858:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828459.924950:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517917568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828459.925013:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828459.925018:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828459.925021:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828459.925044:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1466 00000100:00080000:2.0:1787828459.925059:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828459.925063:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828459.925073:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828459.925087:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517917696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828459.925098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828459.925104:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828459.925107:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828459.925116:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1466 00000100:00080000:2.0:1787828459.925119:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828459.925122:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828459.925126:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828459.925137:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517917824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828459.925146:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828459.925149:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828459.925150:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828459.925153:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1466 00000100:00080000:2.0:1787828459.925169:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828459.925173:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828459.925177:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828459.925186:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517917952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828459.925194:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828459.925197:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828459.925198:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828459.925202:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1466 00000100:00080000:2.0:1787828459.925205:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828459.925208:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828459.925211:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828459.925227:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517918080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828459.925235:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828459.925239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828459.925241:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828459.925250:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1466 00000100:00080000:2.0:1787828459.925256:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828459.925260:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828459.925266:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828459.925279:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517918208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828459.925290:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828459.925295:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828459.925302:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828459.925308:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1466 00000100:00080000:2.0:1787828459.925313:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828459.925317:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828459.925322:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828459.925335:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517918336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828460.756912:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828460.948807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828461.780802:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828461.972756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828462.804778:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828462.996730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:2.0:1787828462.996996:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787828462.997064:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828462.997081:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828462.997117:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828462.997126:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828462.997134:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828462.997143:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828462.997150:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828462.997158:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00010000:00010000:2.0:1787828463.828958:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828464.020729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828464.852875:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828465.044781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828465.044839:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828465.044854:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828465.044860:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828465.044876:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1471 00000100:00080000:2.0:1787828465.044892:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828465.044905:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828465.044969:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517918464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828465.045021:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828465.045026:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828465.045029:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828465.045037:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1471 00000100:00080000:2.0:1787828465.045045:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828465.045051:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828465.045066:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828465.045116:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517918592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828465.045141:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828465.045149:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828465.045155:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828465.045165:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1471 00000100:00080000:2.0:1787828465.045175:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828465.045187:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828465.045204:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828465.045234:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517918720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828465.045278:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828465.045284:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828465.045288:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828465.045295:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1471 00000100:00080000:2.0:1787828465.045302:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828465.045317:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828465.045336:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828465.045367:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517918848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828465.045402:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828465.045415:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828465.045427:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828465.045437:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1471 00000100:00080000:2.0:1787828465.045445:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828465.045454:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828465.045465:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828465.045503:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517918976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828465.045535:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828465.045545:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828465.045549:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828465.045567:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1471 00000100:00080000:2.0:1787828465.045583:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828465.045598:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828465.045617:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828465.045649:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517919104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828465.045673:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828465.045732:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828465.045738:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828465.045749:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1471 00000100:00080000:2.0:1787828465.045760:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828465.045771:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828465.045781:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828465.045848:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517919232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828465.876860:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828466.068739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828466.900823:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828467.092750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828467.924833:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828468.116718:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828468.948789:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828469.140776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828469.972853:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828470.164865:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828470.164873:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828470.164893:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828470.164900:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828470.164913:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1476 00000100:00080000:0.0:1787828470.164929:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828470.164939:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828470.164990:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517919360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828470.165026:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828470.165031:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828470.165034:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828470.165041:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1476 00000100:00080000:0.0:1787828470.165047:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828470.165054:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828470.165068:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828470.165091:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517919488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828470.165128:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828470.165135:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828470.165140:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828470.165149:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1476 00000100:00080000:0.0:1787828470.165157:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828470.165166:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828470.165177:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828470.165206:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517919616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828470.165245:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828470.165251:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828470.165255:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828470.165262:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1476 00000100:00080000:0.0:1787828470.165268:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828470.165274:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828470.165283:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828470.165320:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517919744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828470.165348:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828470.165356:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828470.165363:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828470.165378:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1476 00000100:00080000:0.0:1787828470.165388:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828470.165419:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828470.165428:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828470.165449:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517919872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828470.165467:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828470.165477:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828470.165482:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828470.165491:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1476 00000100:00080000:0.0:1787828470.165503:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828470.165514:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828470.165525:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828470.165551:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517920000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828470.165576:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828470.165581:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828470.165587:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828470.165596:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1476 00000100:00080000:0.0:1787828470.165604:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828470.165616:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828470.165626:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828470.165651:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517920128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828470.996812:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828471.188848:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828472.020922:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828472.212805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828473.044861:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828473.236844:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828473.236860:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828474.068819:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828474.260733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828474.260743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828475.092784:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828475.284841:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828475.284867:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828475.284875:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828475.284880:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828475.284889:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1481 00000100:00080000:2.0:1787828475.284909:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828475.284919:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828475.284959:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517920256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828475.284982:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828475.284985:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828475.284987:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828475.284991:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1481 00000100:00080000:2.0:1787828475.284994:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828475.284997:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828475.285006:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828475.285018:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517920384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828475.285030:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828475.285041:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828475.285043:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828475.285046:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1481 00000100:00080000:2.0:1787828475.285049:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828475.285053:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828475.285057:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828475.285067:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517920512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828475.285076:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828475.285078:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828475.285080:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828475.285083:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1481 00000100:00080000:2.0:1787828475.285088:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828475.285091:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828475.285094:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828475.285107:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517920640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828475.285117:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828475.285120:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828475.285122:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828475.285126:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1481 00000100:00080000:2.0:1787828475.285128:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828475.285131:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828475.285135:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828475.285146:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517920768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828475.285168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828475.285173:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828475.285175:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828475.285183:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1481 00000100:00080000:2.0:1787828475.285185:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828475.285193:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828475.285200:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828475.285210:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517920896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828475.285228:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828475.285232:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828475.285234:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828475.285238:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1481 00000100:00080000:2.0:1787828475.285241:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828475.285244:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828475.285247:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828475.285258:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517921024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828475.285344:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828476.116804:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828476.308722:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828476.308747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828477.140976:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828477.332736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828477.332753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828478.164841:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828478.356876:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828478.356924:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828479.188916:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828479.380821:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828479.380821:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828480.212957:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828480.404859:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828480.404878:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828480.404878:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828480.404897:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828480.404905:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828480.404923:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1486 00000100:00080000:2.0:1787828480.404940:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828480.404954:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828480.405014:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517921152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828480.405063:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828480.405070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828480.405074:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828480.405083:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1486 00000100:00080000:2.0:1787828480.405091:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828480.405100:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828480.405117:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828480.405164:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517921280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828480.405184:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828480.405189:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828480.405192:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828480.405198:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1486 00000100:00080000:2.0:1787828480.405218:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828480.405225:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828480.405234:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828480.405258:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517921408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828480.405312:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828480.405331:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828480.405339:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828480.405353:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1486 00000100:00080000:2.0:1787828480.405364:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828480.405383:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828480.405409:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828480.405458:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517921536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828480.405480:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828480.405487:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828480.405491:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828480.405499:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1486 00000100:00080000:2.0:1787828480.405528:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828480.405538:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828480.405549:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828480.405589:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517921664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828480.405644:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828480.405651:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828480.405654:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828480.405723:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1486 00000100:00080000:2.0:1787828480.405736:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828480.405750:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828480.405761:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828480.405806:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517921792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828480.405822:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828480.405830:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828480.405836:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828480.405869:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1486 00000100:00080000:2.0:1787828480.405880:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828480.405894:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828480.405910:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828480.405940:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517921920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828481.236852:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828481.428805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828481.428805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828482.260900:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828482.452775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828482.452775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828483.284808:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828483.476728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828483.476728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828484.308843:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828484.500735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828484.500739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:2.0:1787828485.332803:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828485.524714:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828485.524718:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828485.524725:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787828485.524728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828485.524730:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828485.524740:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1491 00000100:00080000:2.0:1787828485.524750:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828485.524757:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828485.524793:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517922048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828485.524816:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828485.524819:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828485.524820:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828485.524823:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1491 00000100:00080000:2.0:1787828485.524825:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828485.524828:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828485.524836:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828485.524847:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517922176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828485.524855:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828485.524857:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828485.524859:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828485.524863:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1491 00000100:00080000:2.0:1787828485.524865:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828485.524872:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828485.524876:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828485.524900:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517922304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828485.524910:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828485.524914:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828485.524916:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828485.524919:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1491 00000100:00080000:2.0:1787828485.524934:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828485.524939:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828485.524944:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828485.524973:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517922432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828485.524980:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828485.524983:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828485.524984:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828485.524988:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1491 00000100:00080000:2.0:1787828485.524990:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828485.524993:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828485.524995:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828485.525001:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517922560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828485.525007:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828485.525009:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828485.525011:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828485.525029:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1491 00000100:00080000:2.0:1787828485.525033:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828485.525050:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828485.525059:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828485.525074:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517922688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828485.525118:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828485.525123:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828485.525125:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828485.525132:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1491 00000100:00080000:2.0:1787828485.525136:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828485.525139:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828485.525142:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828485.525149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517922816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828486.356902:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828486.548749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828486.548749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828487.380795:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828487.572722:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828487.572723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828488.404891:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828488.596792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828488.596792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828489.428835:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828489.620727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828489.620750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00010000:00010000:1.0:1787828490.452838:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828490.644846:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828490.644859:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828490.644864:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828490.644877:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828490.644925:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1497 00000100:00080000:0.0:1787828490.644945:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828490.644960:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828490.645020:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517922944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828490.645070:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828490.645078:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828490.645082:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828490.645101:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1497 00000100:00080000:0.0:1787828490.645110:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828490.645136:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828490.645159:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828490.645208:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517923072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828490.645234:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828490.645241:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828490.645245:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828490.645252:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1497 00000100:00080000:0.0:1787828490.645278:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828490.645295:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828490.645306:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828490.645332:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517923200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828490.645444:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828490.645465:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828490.645475:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828490.645498:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1497 00000100:00080000:0.0:1787828490.645514:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828490.645530:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828490.645542:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828490.645585:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517923328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828490.645704:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828490.645712:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828490.645716:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828490.645731:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1497 00000100:00080000:0.0:1787828490.645748:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828490.645765:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828490.645784:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828490.645817:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517923456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828490.645841:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828490.645851:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828490.645859:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828490.645879:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1497 00000100:00080000:0.0:1787828490.645892:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828490.645910:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828490.645945:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828490.645984:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517923584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828490.646015:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828490.646026:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828490.646031:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828490.646043:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1497 00000100:00080000:0.0:1787828490.646053:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828490.646066:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828490.646083:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828490.646116:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517923712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828491.476806:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828491.668741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828492.500820:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828492.692764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828493.524854:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828493.716779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828494.548771:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828494.740763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828495.572800:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828495.764738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828495.764743:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.764750:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828495.764755:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.764764:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1502 00000100:00080000:1.0:1787828495.764785:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.764793:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828495.764834:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517923840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828495.764857:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.764860:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828495.764862:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.764876:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1502 00000100:00080000:1.0:1787828495.764879:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.764883:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828495.764892:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828495.764906:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517923968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828495.764915:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.764917:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828495.764919:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.764921:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1502 00000100:00080000:1.0:1787828495.764924:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.764926:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828495.764941:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828495.764949:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517924096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828495.764956:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.764958:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828495.764960:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.764962:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1502 00000100:00080000:1.0:1787828495.764965:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.764967:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828495.764971:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828495.764989:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517924224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828495.764996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.764999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828495.765000:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.765002:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1502 00000100:00080000:1.0:1787828495.765004:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.765007:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828495.765011:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828495.765021:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517924352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828495.765049:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.765052:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828495.765053:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.765059:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1502 00000100:00080000:1.0:1787828495.765064:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.765070:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828495.765077:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828495.765086:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517924480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828495.765101:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828495.765104:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828495.765106:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828495.765109:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1502 00000100:00080000:1.0:1787828495.765112:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828495.765114:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828495.765116:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828495.765123:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517924608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828496.596872:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828496.788792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828497.620924:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828497.812760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828498.644883:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828498.836815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828499.668783:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828499.860789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828500.692933:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828500.884838:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828500.884840:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828500.884855:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828500.884863:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828500.884880:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1507 00000100:00080000:1.0:1787828500.884913:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828500.884926:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828500.884987:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517924736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828500.885034:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828500.885042:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828500.885046:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828500.885055:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1507 00000100:00080000:1.0:1787828500.885061:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828500.885067:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828500.885083:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828500.885131:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517924864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828500.885148:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828500.885153:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828500.885156:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828500.885160:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1507 00000100:00080000:1.0:1787828500.885164:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828500.885177:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828500.885183:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828500.885197:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517924992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828500.885208:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828500.885212:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828500.885214:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828500.885218:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1507 00000100:00080000:1.0:1787828500.885222:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828500.885225:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828500.885231:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828500.885250:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517925120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828500.885267:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828500.885271:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828500.885274:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828500.885280:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1507 00000100:00080000:1.0:1787828500.885295:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828500.885299:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828500.885305:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828500.885317:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517925248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828500.885326:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828500.885330:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828500.885333:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828500.885346:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1507 00000100:00080000:1.0:1787828500.885357:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828500.885376:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828500.885389:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828500.885404:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517925376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828500.885420:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828500.885434:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828500.885443:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828500.885470:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1507 00000100:00080000:1.0:1787828500.885484:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828500.885497:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828500.885503:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828500.885526:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517925504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828501.716856:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828501.908842:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828502.740836:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828502.932726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828503.764845:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828503.956794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828504.788921:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828504.980813:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828505.812827:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828506.004793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828506.005199:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517925632:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787828506.005290:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828506.005357:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828506/real 1787828506] req@ffff8b2e8803b480 x1874672517925632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828506.005410:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517925632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828506.005431:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828506.005447:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828506.005460:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517925632:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828506.005534:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517925760:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828506.005587:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828506.005615:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828506/real 1787828506] req@ffff8b2e8803a1c0 x1874672517925760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828506.005683:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517925760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828506.005698:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828506.005716:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828506.005725:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517925760:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828506.005744:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517925888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828506.005784:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828506.005807:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828506/real 1787828506] req@ffff8b2e88039e00 x1874672517925888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828506.005851:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517925888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828506.005882:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828506.005897:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828506.005905:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517925888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828506.005925:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517926016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828506.005962:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828506.005985:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828506/real 1787828506] req@ffff8b2e88039680 x1874672517926016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828506.006001:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517926016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828506.006026:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828506.006030:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828506.006034:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517926016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828506.006054:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517926144:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828506.006078:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828506.006090:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828506/real 1787828506] req@ffff8b2e8803b480 x1874672517926144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828506.006105:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517926144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828506.006115:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828506.006119:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828506.006123:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517926144:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828506.006130:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517926272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828506.006148:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828506.006159:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828506/real 1787828506] req@ffff8b2e880392c0 x1874672517926272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828506.006173:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517926272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828506.006191:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828506.006195:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828506.006199:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517926272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828506.006205:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517926400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828506.006222:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828506.006232:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828506/real 1787828506] req@ffff8b2e880383c0 x1874672517926400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828506.006245:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517926400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828531 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828506.006255:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828506.006258:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828506.006262:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517926400:192.168.203.101@tcp:8:kworker.0 00010000:00010000:2.0:1787828506.836784:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828507.028756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828507.860891:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828508.052770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828508.884918:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828509.076835:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828509.908928:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828510.100831:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828510.933030:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828511.124775:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828511.124781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828511.124787:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828511.124794:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828511.124805:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1517 00000100:00080000:1.0:1787828511.124817:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828511.124825:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828511.124861:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517926528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828511.124891:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828511.124895:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828511.124897:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828511.124901:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1517 00000100:00080000:1.0:1787828511.124906:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828511.124910:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828511.124921:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828511.124929:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517926528:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828511.124937:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517926656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828511.124960:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828511.124966:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828511.124968:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828511.124974:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1517 00000100:00080000:1.0:1787828511.124977:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828511.124982:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828511.124987:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828511.124998:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517926784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828511.125009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828511.125012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828511.125015:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828511.125020:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1517 00000100:00080000:1.0:1787828511.125024:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828511.125028:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828511.125034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828511.125043:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828511.125054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517926912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828511.125068:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828511.125072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828511.125074:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828511.125080:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1517 00000100:00080000:1.0:1787828511.125084:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828511.125089:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828511.125095:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828511.125103:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828511/real 1787828511] req@ffff8b2e88077840 x1874672517926528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828511.125114:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517927040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828511.125142:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828511.125146:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828511.125148:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517926528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828511.125151:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828511.125167:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1517 00000100:00080000:0.0:1787828511.125171:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828511.125174:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828511.125178:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828511.125186:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517926528:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828511.125186:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828511.125206:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828511.125224:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517927168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828511.125228:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517926656:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828511.125255:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828511.125259:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828511.125262:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828511.125271:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1517 00000800:00000100:0.0:1787828511.125272:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828511.125284:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828511.125290:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828511.125294:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828511/real 1787828511] req@ffff8b2e88074780 x1874672517926656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828511.125301:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828511.125312:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517926656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828511.125321:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828511.125323:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517927296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828511.125328:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828511.125337:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517926656:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828511.125355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517926784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828511.125379:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828511.125388:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828511/real 1787828511] req@ffff8b2e88076940 x1874672517926784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828511.125399:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517926784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828511.125406:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828511.125409:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828511.125412:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517926784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828511.125416:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517926912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828511.125427:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828511.125435:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828511/real 1787828511] req@ffff8b2e88074f00 x1874672517926912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828511.125444:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517926912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828511.125450:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828511.125452:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828511.125455:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517926912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828511.125460:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517927040:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828511.125496:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828511.125505:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828511/real 1787828511] req@ffff8b2e88075680 x1874672517927040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828511.125515:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517927040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828511.125523:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828511.125526:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828511.125528:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517927040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828511.125544:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517927168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828511.125557:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828511.125564:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828511/real 1787828511] req@ffff8b2e88075a40 x1874672517927168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828511.125574:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517927168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828511.125581:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828511.125583:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828511.125586:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517927168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828511.125590:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517927296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828511.125601:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828511.125607:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828511/real 1787828511] req@ffff8b2e88077480 x1874672517927296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828511.125616:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517927296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828536 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828511.125622:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828511.125624:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828511.125627:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517927296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828512.148738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828512.980924:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828513.172766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828514.004787:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828514.196722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828515.028813:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828515.220755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828516.052803:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828516.244798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828516.244841:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828516.244859:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828516.244868:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828516.244891:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1522 00000100:00080000:1.0:1787828516.244923:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828516.244940:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828516.245023:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517927424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828516.245080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828516.245088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828516.245092:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828516.245102:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1522 00000100:00080000:1.0:1787828516.245120:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828516.245130:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828516.245165:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828516.245172:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517927424:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828516.245194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517927552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828516.245213:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828516.245219:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828516.245222:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828516.245237:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1522 00000100:00080000:1.0:1787828516.245268:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828516.245275:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787828516.245292:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787828516.245297:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828516.245331:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517927680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828516.245348:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828516.245353:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828516.245358:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828516.245367:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1522 00000100:00080000:1.0:1787828516.245372:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828516.245390:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828516.245398:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828516.245399:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff8b2e88077c00 x1874672517927424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828516.245423:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517927808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828516.245466:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517927424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828516.245487:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828516.245510:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828516.245521:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828516.245535:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828516.245536:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1522 00000100:00080000:1.0:1787828516.245555:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828516.245579:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828516.245580:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787828516.245595:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828516.245622:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517927424:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828516.245633:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517927936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828516.245672:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517927552:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828516.245699:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828516.245705:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828516.245709:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828516.245717:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1522 00000100:00080000:1.0:1787828516.245724:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828516.245730:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828516.245737:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828516.245765:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517928064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828516.245779:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828516.245785:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828516.245795:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828516.245797:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828516.245810:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1522 00000100:00080000:1.0:1787828516.245827:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828516.245832:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828516.245839:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828516.245856:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff8b2e88077480 x1874672517927552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828516.245859:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517928192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828516.245925:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517927552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828516.245942:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828516.245965:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828516.245970:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517927552:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828516.245981:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517927680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828516.246009:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828516.246042:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff8b2e88075a40 x1874672517927680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828516.246055:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517927680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828516.246065:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828516.246069:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828516.246072:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517927680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828516.246098:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517927808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828516.246125:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828516.246143:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff8b2e88075680 x1874672517927808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828516.246177:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517927808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828516.246189:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828516.246194:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828516.246199:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517927808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828516.246219:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517927936:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828516.246240:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828516.246270:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff8b2e88074f00 x1874672517927936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828516.246283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517927936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828516.246294:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828516.246297:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828516.246300:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517927936:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828516.246324:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517928064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828516.246345:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828516.246356:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff8b2e88076940 x1874672517928064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828516.246370:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517928064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828516.246397:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828516.246400:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828516.246403:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517928064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828516.246410:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517928192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828516.246456:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828516.246490:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828516/real 1787828516] req@ffff8b2e88074780 x1874672517928192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828516.246506:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517928192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828541 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828516.246517:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828516.246521:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828516.246525:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517928192:192.168.203.101@tcp:8:kworker.0 00010000:00010000:2.0:1787828517.076874:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828517.268800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828518.100963:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828518.292821:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828519.124827:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828519.316779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828520.148891:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828520.340784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828521.172826:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828521.364762:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828521.364769:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787828521.364770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828521.364774:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828521.364786:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1527 00000100:00080000:0.0:1787828521.364817:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828521.364827:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828521.364895:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517928320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828521.364923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828521.364927:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828521.364928:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828521.364932:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1527 00000100:00080000:0.0:1787828521.364937:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828521.364941:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828521.364953:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828521.364968:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517928448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828521.364978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828521.364981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828521.364983:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828521.364986:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1527 00000100:00080000:0.0:1787828521.364989:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828521.364991:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828521.364994:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828521.365005:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517928576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828521.365012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828521.365015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828521.365017:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828521.365020:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1527 00000100:00080000:0.0:1787828521.365031:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828521.365034:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828521.365038:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828521.365052:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517928704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828521.365061:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828521.365064:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828521.365066:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828521.365069:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1527 00000100:00080000:0.0:1787828521.365071:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828521.365074:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828521.365077:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828521.365088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517928832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828521.365102:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828521.365112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828521.365115:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828521.365119:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1527 00000100:00080000:0.0:1787828521.365122:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828521.365129:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828521.365155:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828521.365173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517928960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828521.365185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828521.365188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828521.365191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828521.365200:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1527 00000100:00080000:0.0:1787828521.365222:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828521.365227:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828521.365234:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828521.365259:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517929088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:2.0:1787828522.196983:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828522.388751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828523.220811:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828523.412798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828524.244981:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828524.436803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828525.268872:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828525.460787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828526.292997:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828526.484722:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828526.484726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828526.484735:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828526.484741:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828526.484751:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1532 00000100:00080000:0.0:1787828526.484761:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828526.484768:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828526.484809:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517929216/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828526.484832:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828526.484835:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828526.484837:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828526.484840:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1532 00000100:00080000:0.0:1787828526.484843:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828526.484846:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828526.484854:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828526.484869:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517929344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828526.484880:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828526.484882:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828526.484883:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828526.484885:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1532 00000100:00080000:0.0:1787828526.484888:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828526.484890:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828526.484893:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828526.484901:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517929472/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828526.484907:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828526.484910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828526.484911:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828526.484913:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1532 00000100:00080000:0.0:1787828526.484916:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828526.484918:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828526.484922:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828526.484929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517929600/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828526.484939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828526.484942:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828526.484943:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828526.484947:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1532 00000100:00080000:0.0:1787828526.484949:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828526.484952:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828526.484955:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828526.484963:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517929728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828526.484971:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828526.484978:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828526.484979:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828526.484983:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1532 00000100:00080000:0.0:1787828526.484986:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828526.484989:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828526.484993:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828526.485005:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517929856/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828526.485013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828526.485016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828526.485018:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828526.485023:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1532 00000100:00080000:0.0:1787828526.485030:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828526.485034:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828526.485038:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828526.485047:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517929984/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828527.508817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828528.340830:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828528.532865:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828529.364924:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828529.556948:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:0.0:1787828529.557407:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:0.0:1787828529.557480:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828529.557514:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828529.557575:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828529.557586:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828529.557595:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828529.557604:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828529.557614:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828529.557627:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787828530.580802:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828531.412847:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828531.604841:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.604854:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828531.604860:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.604886:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1538 00000100:00080000:0.0:1787828531.604901:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.604914:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828531.604956:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828531.604990:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517930112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828531.605037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.605044:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828531.605047:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.605055:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1538 00000100:00080000:0.0:1787828531.605061:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.605068:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828531.605096:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828531.605118:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517930240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828531.605136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.605141:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828531.605144:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.605149:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1538 00000100:00080000:0.0:1787828531.605154:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.605160:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828531.605168:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828531.605183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517930368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828531.605199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.605204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828531.605207:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.605213:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1538 00000100:00080000:0.0:1787828531.605231:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.605239:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828531.605247:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828531.605264:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517930496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828531.605295:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.605300:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828531.605303:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.605309:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1538 00000100:00080000:0.0:1787828531.605314:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.605341:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828531.605350:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828531.605366:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517930624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828531.605379:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.605384:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828531.605387:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.605392:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1538 00000100:00080000:0.0:1787828531.605397:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.605402:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828531.605409:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828531.605423:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517930752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828531.605439:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828531.605444:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828531.605447:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828531.605453:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1538 00000100:00080000:0.0:1787828531.605459:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828531.605476:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828531.605483:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828531.605498:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517930880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00010000:00010000:1.0:1787828532.436913:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828532.628912:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828533.460882:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828533.652751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:2.0:1787828534.484896:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828534.676756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828535.508918:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828535.700704:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00010000:00010000:1.0:1787828536.532901:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828536.724814:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828536.724815:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.724830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828536.724836:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.724853:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1543 00000100:00080000:0.0:1787828536.724866:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.724876:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828536.724918:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517931008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828536.724947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.724951:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828536.724953:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.724957:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1543 00000100:00080000:0.0:1787828536.724961:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.724967:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828536.724979:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828536.724999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517931136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828536.725039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.725042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828536.725046:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.725052:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1543 00000100:00080000:0.0:1787828536.725057:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.725061:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828536.725067:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828536.725086:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517931264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828536.725095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.725099:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828536.725101:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.725105:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1543 00000100:00080000:0.0:1787828536.725111:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.725116:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828536.725120:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828536.725133:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517931392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828536.725140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.725144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828536.725146:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.725155:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1543 00000100:00080000:0.0:1787828536.725164:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.725172:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828536.725178:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828536.725193:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517931520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828536.725203:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.725207:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828536.725208:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.725214:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1543 00000100:00080000:0.0:1787828536.725220:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.725228:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828536.725236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828536.725257:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517931648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828536.725278:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828536.725281:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828536.725305:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828536.725313:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1543 00000100:00080000:0.0:1787828536.725322:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828536.725331:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828536.725352:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828536.725382:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517931776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828537.556852:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828537.556853:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828537.748788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828538.580776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828538.580864:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828538.772782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828539.604749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828539.604786:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828539.796775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828540.628750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828540.628820:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828540.820821:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828541.652789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828541.652975:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828541.844794:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828541.844796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828541.844818:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828541.844825:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828541.844843:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1548 00000100:00080000:2.0:1787828541.844859:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828541.844873:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828541.844929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517931904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828541.844985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828541.844990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828541.844993:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828541.844999:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1548 00000100:00080000:2.0:1787828541.845004:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828541.845008:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828541.845020:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828541.845036:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517932032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828541.845049:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828541.845054:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828541.845056:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828541.845060:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1548 00000100:00080000:2.0:1787828541.845065:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828541.845069:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828541.845076:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828541.845091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517932160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828541.845113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828541.845117:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828541.845120:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828541.845124:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1548 00000100:00080000:2.0:1787828541.845128:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828541.845149:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828541.845155:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828541.845174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517932288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828541.845187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828541.845192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828541.845195:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828541.845201:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1548 00000100:00080000:2.0:1787828541.845206:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828541.845215:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828541.845222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828541.845241:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517932416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828541.845277:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828541.845282:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828541.845284:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828541.845289:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1548 00000100:00080000:2.0:1787828541.845295:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828541.845299:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828541.845306:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828541.845333:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517932544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828541.845389:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828541.845396:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828541.845399:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828541.845405:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1548 00000100:00080000:2.0:1787828541.845410:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828541.845420:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828541.845432:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828541.845453:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517932672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828542.676786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828542.676798:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828542.868791:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828543.700787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828543.700898:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828543.892751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828544.724772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828544.724852:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828544.916733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828545.748752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828545.940740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828546.772794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828546.772911:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828546.964802:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828546.964846:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.964861:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828546.964870:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.964889:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1553 00000100:00080000:1.0:1787828546.964921:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.964935:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828546.965018:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517932800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.965082:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.965088:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828546.965091:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.965097:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1553 00000100:00080000:1.0:1787828546.965104:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.965112:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828546.965129:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828546.965154:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517932928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.965175:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.965180:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828546.965183:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.965189:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1553 00000100:00080000:1.0:1787828546.965194:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.965198:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828546.965204:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828546.965257:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517933056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.965272:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.965286:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828546.965290:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.965295:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1553 00000100:00080000:1.0:1787828546.965301:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.965307:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828546.965322:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828546.965341:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517933184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.965355:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.965359:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828546.965362:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.965367:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1553 00000100:00080000:1.0:1787828546.965371:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.965375:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828546.965382:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828546.965395:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517933312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.965407:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.965412:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828546.965420:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.965431:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1553 00000100:00080000:1.0:1787828546.965439:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.965449:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828546.965461:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828546.965488:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517933440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828546.965509:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828546.965523:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828546.965529:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828546.965542:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1553 00000100:00080000:1.0:1787828546.965548:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828546.965563:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828546.965576:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828546.965595:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517933568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828547.796755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828547.796828:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828547.988817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828548.820808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828549.012829:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828549.844796:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828550.036736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828550.868792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828550.868857:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828551.060788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828551.892859:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828552.084798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828552.084893:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828552.084907:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828552.084914:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828552.084928:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1558 00000100:00080000:2.0:1787828552.084943:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828552.084952:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828552.085010:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517933696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828552.085066:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828552.085071:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828552.085075:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828552.085080:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1558 00000100:00080000:2.0:1787828552.085087:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828552.085093:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828552.085107:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828552.085136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517933824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828552.085151:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828552.085159:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828552.085167:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828552.085223:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1558 00000100:00080000:2.0:1787828552.085231:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828552.085240:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828552.085251:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828552.085288:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517933952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828552.085309:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828552.085314:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828552.085318:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828552.085323:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1558 00000100:00080000:2.0:1787828552.085361:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828552.085367:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828552.085374:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828552.085397:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517934080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828552.085413:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828552.085417:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828552.085420:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828552.085426:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1558 00000100:00080000:2.0:1787828552.085431:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828552.085435:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828552.085441:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828552.085459:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517934208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828552.085547:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828552.085560:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828552.085563:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828552.085571:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1558 00000100:00080000:2.0:1787828552.085577:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828552.085587:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828552.085596:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828552.085617:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517934336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828552.085703:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828552.085715:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828552.085729:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828552.085742:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1558 00000100:00080000:2.0:1787828552.085746:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828552.085751:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828552.085758:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828552.085778:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517934464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828552.916778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828553.108807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828553.940849:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828554.132783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828554.964785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828555.156820:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828555.988776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828556.180815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828557.012722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828557.204780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828557.204823:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828557.204830:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828557.204836:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828557.204849:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1563 00000100:00080000:1.0:1787828557.204863:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828557.204871:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828557.204920:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517934592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828557.204954:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828557.204958:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828557.204960:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828557.204964:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1563 00000100:00080000:1.0:1787828557.204968:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828557.204972:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828557.204985:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828557.205012:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517934720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828557.205024:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828557.205028:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828557.205030:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828557.205032:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1563 00000100:00080000:1.0:1787828557.205035:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828557.205037:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828557.205041:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828557.205052:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517934848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828557.205060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828557.205063:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828557.205064:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828557.205066:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1563 00000100:00080000:1.0:1787828557.205071:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828557.205074:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828557.205078:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828557.205098:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517934976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828557.205106:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828557.205109:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828557.205111:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828557.205114:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1563 00000100:00080000:1.0:1787828557.205118:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828557.205122:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828557.205131:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828557.205161:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517935104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828557.205170:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828557.205174:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828557.205176:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828557.205179:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1563 00000100:00080000:1.0:1787828557.205183:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828557.205186:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828557.205191:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828557.205212:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517935232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828557.205247:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828557.205252:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828557.205254:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828557.205258:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1563 00000100:00080000:1.0:1787828557.205264:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828557.205273:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828557.205281:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828557.205295:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517935360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787828558.036744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828558.228792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828559.060797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828559.252841:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828560.084813:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828560.276863:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828561.108783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828561.300798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828562.132764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:0.0:1787828562.324751:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828562.324762:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828562.324766:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828562.324773:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1568 00000100:00100000:1.0:1787828562.324781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828562.324783:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828562.324792:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828562.324831:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517935488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828562.324860:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828562.324864:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828562.324866:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828562.324872:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1568 00000100:00080000:0.0:1787828562.324876:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828562.324879:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828562.324888:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828562.324898:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517935616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828562.324909:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828562.324913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828562.324916:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828562.324920:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1568 00000100:00080000:0.0:1787828562.324924:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828562.324928:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828562.324935:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828562.324960:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517935744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828562.324972:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828562.324976:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828562.324978:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828562.324982:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1568 00000100:00080000:0.0:1787828562.324987:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828562.324992:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828562.324997:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828562.325011:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517935872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828562.325025:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828562.325027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828562.325029:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828562.325033:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1568 00000100:00080000:0.0:1787828562.325043:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828562.325047:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828562.325051:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828562.325062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517936000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828562.325097:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828562.325099:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828562.325100:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828562.325120:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1568 00000100:00080000:0.0:1787828562.325125:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828562.325131:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828562.325138:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828562.325150:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517936128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828562.325166:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828562.325172:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828562.325173:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828562.325177:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1568 00000100:00080000:0.0:1787828562.325179:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828562.325181:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828562.325185:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828562.325195:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517936256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828562.325250:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517935744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828562.325265:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828562.325272:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff8b2e88040f00 x1874672517935744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828562.325279:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517935744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828562.325285:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828562.325287:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828562.325289:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517935744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828562.325293:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517935872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828562.325301:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828562.325307:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff8b2e88042940 x1874672517935872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828562.325314:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517935872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828562.325319:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828562.325321:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828562.325322:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517935872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828562.325326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517936000:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828562.325335:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828562.325340:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff8b2e88043480 x1874672517936000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828562.325352:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517936000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828562.325357:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828562.325359:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828562.325361:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517936000:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828562.325364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517936128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828562.325373:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828562.325379:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff8b2e880430c0 x1874672517936128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828562.325386:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517936128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828562.325391:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828562.325393:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828562.325394:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517936128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828562.325397:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517936256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828562.325406:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828562.325411:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828562/real 1787828562] req@ffff8b2e88040000 x1874672517936256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828562.325418:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517936256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828587 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828562.325426:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828562.325428:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828562.325430:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517936256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828562.644719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828563.156773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828563.156837:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828563.348713:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828563.348742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828563.668772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828564.180751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828564.180840:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828564.372748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828564.372752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828564.692733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828565.204739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828565.204887:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828565.396766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828565.396784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828565.716762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828566.228816:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828566.228977:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828566.420766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828566.420801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828566.740779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828567.252721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828567.252846:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828567.444734:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828567.444734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828567.444745:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828567.444750:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828567.444754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828567.444760:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1573 00000100:00080000:0.0:1787828567.444772:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.444779:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828567.444814:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517936384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828567.444839:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.444842:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828567.444843:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.444846:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1573 00000100:00080000:0.0:1787828567.444848:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.444851:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828567.444860:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828567.444887:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517936512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828567.444890:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517936384:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828567.444907:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.444912:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828567.444914:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.444920:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1573 00000100:00080000:0.0:1787828567.444925:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.444930:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828567.444937:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828567.444952:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517936640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828567.444961:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828567.444966:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828567.444968:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828567.444971:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.444975:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1573 00000100:00080000:0.0:1787828567.444979:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.444983:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828567.444988:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828567.444998:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517936768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828567.445005:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.445008:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828567.445010:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828567.445015:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828567] req@ffff8b2e880403c0 x1874672517936384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828567.445017:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1573 00000100:00080000:0.0:1787828567.445023:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.445030:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828567.445037:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828567.445043:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672517936384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828567.445048:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517936896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828567.445067:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.445087:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828567.445088:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828567.445090:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828567.445095:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787828567.445097:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1573 00000100:00100000:3.0:1787828567.445102:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517936384:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828567.445103:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.445112:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828567.445117:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828567.445126:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517937024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828567.445133:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517936512:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828567.445138:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828567.445141:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828567.445143:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828567.445146:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1573 00000100:00080000:0.0:1787828567.445149:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828567.445151:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828567.445154:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828567.445168:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828567.445180:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517937152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828567.445188:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828567] req@ffff8b2e88040000 x1874672517936512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828567.445208:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517936512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828567.445218:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828567.445223:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828567.445227:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517936512:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828567.445233:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517936640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828567.445256:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828567.445269:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828567] req@ffff8b2e880430c0 x1874672517936640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828567.445281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517936640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828567.445290:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828567.445294:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828567.445298:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517936640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828567.445305:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517936768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828567.445322:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828567.445330:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828567] req@ffff8b2e88043480 x1874672517936768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828567.445339:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517936768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828567.445345:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828567.445347:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828567.445349:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517936768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828567.445353:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517936896:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828567.445365:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828567.445375:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828567] req@ffff8b2e88042940 x1874672517936896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828567.445388:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517936896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828567.445398:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828567.445401:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828567.445405:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517936896:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828567.445425:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517937024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828567.445443:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828567.445452:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828567] req@ffff8b2e88040f00 x1874672517937024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828567.445460:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517937024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828567.445471:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828567.445474:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828567.445476:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517937024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828567.445479:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517937152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828567.445490:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828567.445496:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828567/real 1787828567] req@ffff8b2e88041a40 x1874672517937152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828567.445503:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517937152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828592 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828567.445509:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828567.445510:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828567.445512:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517937152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828567.764732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828568.276766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828568.276956:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828568.468795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828568.468844:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828568.788775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828569.300770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828569.300957:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828569.492755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828569.492796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828569.812738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828570.324743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828570.324780:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828570.516726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828570.516760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828570.836720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828571.348786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828571.348942:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828571.540744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828571.540758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828571.860759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828572.372761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828572.372838:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828572.564731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828572.564735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828572.564780:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828572.564787:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828572.564791:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828572.564800:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1578 00000100:00080000:1.0:1787828572.564811:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828572.564830:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828572.564874:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517937280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828572.564899:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828572.564902:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828572.564903:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828572.564907:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1578 00000100:00080000:1.0:1787828572.564909:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828572.564912:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828572.564922:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828572.564934:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517937408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828572.564934:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517937280:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828572.564948:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828572.564951:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828572.564952:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828572.564954:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1578 00000100:00080000:1.0:1787828572.564956:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828572.564969:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828572.564972:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828572.564980:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517937536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828572.564986:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828572.564988:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828572.564989:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828572.564993:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1578 00000100:00080000:1.0:1787828572.564995:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828572.564998:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828572.565002:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828572.565008:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517937664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828572.565014:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828572.565017:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828572.565018:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828572.565021:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1578 00000100:00080000:1.0:1787828572.565023:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828572.565025:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828572.565027:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828572.565034:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828572.565050:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517937792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828572.565058:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828572.565061:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828572.565062:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828572.565075:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1578 00000100:00080000:1.0:1787828572.565078:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828572.565081:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828572.565085:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828572.565096:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517937920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828572.565103:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828572.565105:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828572.565107:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828572.565110:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1578 00000100:00080000:1.0:1787828572.565114:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828572.565114:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff8b2e88077840 x1874672517937280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828572.565121:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828572.565127:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828572.565138:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517938048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828572.565142:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517937280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828572.565159:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828572.565162:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828572.565166:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517937280:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828572.565188:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517937408:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828572.565223:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828572.565238:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff8b2e88074780 x1874672517937408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828572.565252:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517937408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828572.565259:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828572.565261:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828572.565264:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517937408:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828572.565269:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517937536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828572.565286:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828572.565294:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff8b2e88076940 x1874672517937536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828572.565306:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517937536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828572.565312:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828572.565314:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828572.565316:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517937536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828572.565320:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517937664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828572.565331:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828572.565337:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff8b2e88074f00 x1874672517937664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828572.565345:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517937664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828572.565350:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828572.565352:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828572.565354:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517937664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828572.565357:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517937792:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828572.565368:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828572.565373:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff8b2e88075680 x1874672517937792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828572.565380:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517937792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828572.565391:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828572.565393:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828572.565395:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517937792:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828572.565398:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517937920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828572.565408:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828572.565413:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff8b2e88075a40 x1874672517937920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828572.565420:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517937920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828572.565426:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828572.565428:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828572.565430:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517937920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828572.565433:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517938048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828572.565443:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828572.565448:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828572/real 1787828572] req@ffff8b2e88077480 x1874672517938048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828572.565456:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517938048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828597 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828572.565460:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828572.565462:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828572.565465:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517938048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828572.884824:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787828573.396794:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828573.396828:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787828573.588754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828573.588805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828573.908734:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828574.420741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828574.420826:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828574.612744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828574.612744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828574.932756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828575.444797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828575.444861:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828575.636797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828575.636885:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828575.956769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828576.468730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828576.468796:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828576.660723:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828576.660789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828576.980730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828577.492753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828577.492847:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828577.684727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828577.684803:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828577.684812:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828577.684817:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828577.684829:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1584 00000100:00080000:1.0:1787828577.684840:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828577.684850:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828577.684858:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828577.684891:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517938176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828577.684918:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828577.684922:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828577.684925:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828577.684929:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1584 00000100:00080000:1.0:1787828577.684933:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828577.684937:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828577.684948:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828577.684967:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517938176:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828577.684968:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517938304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828577.684987:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828577.684991:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828577.684992:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828577.684997:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1584 00000100:00080000:1.0:1787828577.685016:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828577.685020:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828577.685025:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828577.685038:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517938432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828577.685048:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828577.685051:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828577.685064:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828577.685064:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828577.685075:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1584 00000100:00080000:1.0:1787828577.685080:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828577.685084:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828577.685090:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828577.685104:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517938560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828577.685114:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828577.685117:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828577.685123:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828577.685124:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff8b2e88077c00 x1874672517938176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828577.685133:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1584 00000100:00080000:1.0:1787828577.685141:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828577.685149:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828577.685154:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828577.685158:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517938176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828577.685169:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517938688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828577.685174:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828577.685187:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828577.685192:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828577.685196:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517938176:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828577.685199:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828577.685207:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828577.685212:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1584 00000100:00080000:1.0:1787828577.685215:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828577.685219:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828577.685222:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828577.685231:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517938304:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828577.685236:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517938816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828577.685248:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828577.685252:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828577.685254:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828577.685258:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1584 00000800:00000100:3.0:1787828577.685264:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828577.685265:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828577.685273:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828577.685278:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828577.685285:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff8b2e88077480 x1874672517938304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828577.685293:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517938944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828577.685348:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517938304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828577.685357:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828577.685362:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828577.685366:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517938304:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828577.685372:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517938432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828577.685407:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828577.685415:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff8b2e88075a40 x1874672517938432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828577.685425:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517938432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828577.685431:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828577.685434:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828577.685436:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517938432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828577.685440:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517938560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828577.685451:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828577.685467:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff8b2e88075680 x1874672517938560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828577.685476:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517938560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828577.685488:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828577.685491:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828577.685493:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517938560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828577.685497:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517938688:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828577.685511:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828577.685518:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff8b2e88074f00 x1874672517938688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828577.685527:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517938688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828577.685534:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828577.685536:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828577.685538:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517938688:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828577.685553:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517938816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828577.685565:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828577.685573:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff8b2e88076940 x1874672517938816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828577.685588:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517938816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828577.685597:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828577.685601:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828577.685606:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517938816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828577.685614:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517938944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828577.685638:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828577.685674:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828577/real 1787828577] req@ffff8b2e88074780 x1874672517938944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828577.685684:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517938944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828602 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828577.685691:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828577.685694:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828577.685696:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517938944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828578.004763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828578.516760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828578.516938:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828578.708778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828578.708791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828579.028828:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828579.540794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828579.540913:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828579.732803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828579.732817:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828580.052802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828580.564818:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828580.564975:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828580.756819:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828580.756819:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828581.076792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828581.588721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828581.588775:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828581.780730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828581.780732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828582.100717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828582.612796:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828582.612873:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828582.804825:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828582.804825:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828582.804850:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828582.804859:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828582.804863:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828582.804876:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1589 00000100:00080000:1.0:1787828582.804887:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828582.804893:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828582.804931:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517939072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828582.804958:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828582.804962:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828582.804963:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828582.804984:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1589 00000100:00080000:1.0:1787828582.804989:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828582.804994:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828582.805005:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828582.805010:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517939072:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828582.805023:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517939200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828582.805040:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828582.805044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828582.805046:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828582.805049:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1589 00000100:00080000:1.0:1787828582.805053:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828582.805058:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828582.805062:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828582.805071:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517939328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828582.805079:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828582.805082:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828582.805083:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828582.805086:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1589 00000100:00080000:1.0:1787828582.805089:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828582.805097:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787828582.805098:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787828582.805107:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828582.805123:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517939456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828582.805132:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828582.805135:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828582.805137:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828582.805142:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1589 00000100:00080000:1.0:1787828582.805147:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828582.805151:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828582.805157:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff8b2e88077840 x1874672517939072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828582.805158:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828582.805177:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517939584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828582.805188:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828582.805190:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517939072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828582.805192:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828582.805197:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828582.805212:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1589 00000100:00080000:3.0:1787828582.805226:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828582.805227:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828582.805240:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787828582.805241:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787828582.805254:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828582.805255:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517939072:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828582.805275:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517939712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828582.805283:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828582.805287:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828582.805290:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828582.805297:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517939200:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828582.805303:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1589 00000100:00080000:1.0:1787828582.805312:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828582.805316:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828582.805323:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828582.805339:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517939840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828582.805339:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828582.805358:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff8b2e880770c0 x1874672517939200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828582.805373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517939200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.805388:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828582.805394:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828582.805401:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517939200:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828582.805424:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517939328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828582.805446:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828582.805456:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff8b2e88076580 x1874672517939328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828582.805464:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517939328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.805475:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828582.805478:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828582.805480:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517939328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828582.805484:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517939456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828582.805495:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828582.805501:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff8b2e880761c0 x1874672517939456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828582.805508:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517939456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.805525:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828582.805527:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828582.805529:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517939456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828582.805533:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517939584:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828582.805550:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828582.805561:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff8b2e88075e00 x1874672517939584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828582.805585:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517939584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.805595:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828582.805597:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828582.805600:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517939584:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828582.805605:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517939712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828582.805622:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828582.805630:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff8b2e88074b40 x1874672517939712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828582.805638:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517939712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.805644:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828582.805667:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828582.805681:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517939712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828582.805697:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517939840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828582.805715:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828582.805727:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828582/real 1787828582] req@ffff8b2e88074000 x1874672517939840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828582.805749:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517939840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828607 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828582.805759:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828582.805763:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828582.805766:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517939840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828583.124708:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828583.636759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828583.636826:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828583.828793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828583.828793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828584.148727:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828584.660761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828584.660852:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828584.852762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828584.852799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828585.172707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828585.684808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828585.684859:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828585.876751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828585.876751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828586.196744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828586.708781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828586.708799:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828586.900755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828586.900793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828587.220743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828587.732800:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828587.732871:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828587.924754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828587.924773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828587.924806:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828587.924812:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828587.924817:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828587.924827:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1594 00000100:00080000:2.0:1787828587.924839:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828587.924846:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828587.924886:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517939968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828587.924910:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828587.924913:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828587.924914:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828587.924918:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1594 00000100:00080000:2.0:1787828587.924921:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828587.924926:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828587.924948:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828587.924955:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517939968:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828587.924960:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517940096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828587.924969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828587.924972:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828587.924974:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828587.924977:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1594 00000100:00080000:2.0:1787828587.924979:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828587.924982:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828587.924985:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828587.924994:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517940224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828587.925001:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828587.925004:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828587.925005:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828587.925007:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1594 00000100:00080000:2.0:1787828587.925013:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828587.925015:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828587.925019:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828587.925026:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517940352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828587.925031:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828587.925037:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828587.925041:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828587.925043:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828587.925047:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1594 00000100:00080000:2.0:1787828587.925050:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828587.925053:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828587.925056:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828587.925063:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517940480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828587.925070:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828587.925072:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828587.925074:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828587.925077:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1594 00000100:00080000:2.0:1787828587.925080:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828587.925082:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff8b2e8801da40 x1874672517939968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828587.925085:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828587.925091:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828587.925102:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517940608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828587.925108:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828587.925112:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672517939968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828587.925113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828587.925116:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828587.925123:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1594 00000100:00080000:3.0:1787828587.925133:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828587.925134:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828587.925141:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787828587.925144:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:2.0:1787828587.925147:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828587.925152:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517939968:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828587.925158:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517940736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828587.925186:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517940096:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828587.925214:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828587.925229:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff8b2e8801fc00 x1874672517940096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828587.925250:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517940096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828587.925260:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828587.925263:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828587.925267:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517940096:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828587.925273:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517940224:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828587.925295:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828587.925306:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff8b2e8801f480 x1874672517940224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828587.925319:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517940224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828587.925328:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828587.925331:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828587.925335:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517940224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828587.925341:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517940352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828587.925360:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828587.925372:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff8b2e8801cf00 x1874672517940352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828587.925383:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672517940352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828587.925397:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828587.925400:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828587.925404:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517940352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828587.925410:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517940480:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828587.925430:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828587.925442:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff8b2e8801c000 x1874672517940480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828587.925454:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672517940480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828587.925463:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828587.925466:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828587.925470:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517940480:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828587.925477:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517940608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828587.925496:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828587.925508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff8b2e8801e580 x1874672517940608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828587.925521:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517940608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828587.925529:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828587.925532:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828587.925537:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517940608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828587.925543:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517940736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828587.925562:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828587.925574:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828587/real 1787828587] req@ffff8b2e8801e940 x1874672517940736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828587.925587:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517940736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828612 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828587.925597:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828587.925600:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828587.925603:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517940736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828588.244775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828588.756812:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828588.756883:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828588.948719:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828588.948731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828589.268730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828589.780750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828589.781004:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828589.972764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828589.972764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828590.292747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828590.804789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828590.804821:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828590.996711:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828590.996829:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828591.316775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828591.828773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828591.828865:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828592.020722:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828592.020777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828592.340707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828592.852739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828592.852817:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828593.044729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828593.044739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828593.044779:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828593.044785:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828593.044789:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828593.044797:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1599 00000100:00080000:1.0:1787828593.044807:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828593.044812:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828593.044847:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517940864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828593.044869:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828593.044872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828593.044873:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828593.044876:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1599 00000100:00080000:1.0:1787828593.044878:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828593.044881:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828593.044889:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828593.044919:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517940864:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828593.044920:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517940992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828593.044934:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828593.044937:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828593.044938:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828593.044942:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1599 00000100:00080000:1.0:1787828593.044945:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828593.044948:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828593.044952:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828593.044959:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517941120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828593.044966:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828593.044968:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828593.044969:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828593.044972:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1599 00000100:00080000:1.0:1787828593.044974:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828593.044976:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828593.044979:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828593.044985:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517941248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828593.044991:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828593.044993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828593.044994:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828593.044997:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1599 00000100:00080000:1.0:1787828593.044999:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828593.045001:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828593.045004:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828593.045010:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517941376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828593.045018:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828593.045020:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828593.045022:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828593.045025:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1599 00000100:00080000:1.0:1787828593.045027:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828593.045029:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828593.045032:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828593.045038:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517940992:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828593.045044:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517941504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828593.045053:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828593.045054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828593.045056:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828593.045058:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1599 00000100:00080000:1.0:1787828593.045062:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828593.045065:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828593.045068:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828593.045077:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517941120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828593.045078:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517941632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828593.045113:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517941248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828593.045140:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517941376:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828593.045163:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517941504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828593.045185:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517941632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828593.364751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828593.876746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828593.876808:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828594.068723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828594.068746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828594.388730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828594.900743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828594.900810:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828595.092786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828595.092794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828595.412768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828595.924748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828595.924821:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828596.116737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828596.116742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:3.0:1787828596.116964:0:2336:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:3.0:1787828596.117018:0:2336:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828596.117042:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787828596.117103:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787828596.117116:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787828596.117125:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787828596.117127:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828593/real 1787828596] req@ffff8b2e88074000 x1874672517940992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828596.117133:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787828596.117153:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787828596.117156:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517940992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787828596.117162:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00080000:0.0:1787828596.117172:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828596.117180:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828596.117185:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517940992:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828596.117194:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828593/real 1787828596] req@ffff8b2e88074b40 x1874672517941120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828596.117217:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517941120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828596.117227:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828596.117232:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828596.117236:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517941120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828596.117245:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828593/real 1787828596] req@ffff8b2e88075e00 x1874672517941248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828596.117256:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517941248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828596.117263:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828596.117267:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828596.117271:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517941248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828596.117278:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828593/real 1787828596] req@ffff8b2e880761c0 x1874672517941376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828596.117291:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517941376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828596.117303:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828596.117307:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828596.117310:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517941376:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828596.117317:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828593/real 1787828596] req@ffff8b2e88076d00 x1874672517940864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828596.117332:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517940864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828596.117339:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828596.117344:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828596.117347:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517940864:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828596.117384:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828593/real 1787828596] req@ffff8b2e88076580 x1874672517941504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828596.117396:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517941504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828596.117404:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828596.117407:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828596.117411:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517941504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828596.117417:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828593/real 1787828596] req@ffff8b2e880770c0 x1874672517941632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828596.117428:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517941632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828618 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828596.117436:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828596.117440:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828596.117444:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517941632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828596.436799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828596.948746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828596.948775:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828597.140759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828597.140759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828597.460720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828597.972738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828597.972810:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828598.164730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828598.164743:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828598.164750:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828598.164755:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828598.164766:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1604 00000100:00100000:3.0:1787828598.164769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828598.164781:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828598.164790:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828598.164834:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517941760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828598.164854:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828598.164872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828598.164874:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828598.164878:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1604 00000100:00080000:1.0:1787828598.164882:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828598.164886:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828598.164897:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517941760:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787828598.164897:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828598.164921:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517941888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828598.164931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828598.164934:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828598.164935:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828598.164939:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1604 00000100:00080000:1.0:1787828598.164956:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828598.164960:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787828598.164965:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787828598.164966:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828598.164988:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517942016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828598.165000:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828598.165004:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828598.165006:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828598.165011:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1604 00000100:00100000:0.0:1787828598.165016:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828598/real 1787828598] req@ffff8b2e88077840 x1874672517941760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828598.165018:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828598.165033:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828598.165041:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828598.165043:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517941760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828598.165060:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828598.165069:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787828598.165072:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517942144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828598.165075:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517941760:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828598.165100:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828598.165104:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828598.165107:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828598.165110:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517941888:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828598.165117:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1604 00000100:00080000:1.0:1787828598.165124:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828598.165140:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787828598.165147:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787828598.165148:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828598.165171:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517942272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828598.165176:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828598/real 1787828598] req@ffff8b2e88074780 x1874672517941888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828598.165215:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828598.165224:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828598.165225:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517941888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828598.165232:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828598.165246:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1604 00000100:00080000:0.0:1787828598.165251:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828598.165254:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828598.165262:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828598.165269:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828598.165277:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517941888:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787828598.165282:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828598.165293:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517942016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828598.165306:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517942400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828598.165320:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828598.165328:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787828598.165329:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828598.165332:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828598.165344:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1604 00000100:00080000:1.0:1787828598.165350:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828598.165355:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828598/real 1787828598] req@ffff8b2e88076940 x1874672517942016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828598.165362:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828598.165374:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828598.165396:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517942528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828598.165403:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517942016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828598.165450:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828598.165456:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828598.165460:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517942016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828598.165469:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517942144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828598.165498:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828598.165511:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828598/real 1787828598] req@ffff8b2e88074f00 x1874672517942144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828598.165524:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517942144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828598.165532:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828598.165537:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828598.165540:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517942144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828598.165560:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517942272:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828598.165582:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828598.165594:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828598/real 1787828598] req@ffff8b2e88075680 x1874672517942272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828598.165605:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517942272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828598.165627:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828598.165632:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828598.165635:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517942272:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828598.165642:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517942400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828598.165697:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828598.165709:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828598/real 1787828598] req@ffff8b2e88075a40 x1874672517942400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828598.165721:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517942400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828598.165730:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828598.165733:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828598.165736:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517942400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828598.165744:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517942528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828598.165765:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828598.165787:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828598/real 1787828598] req@ffff8b2e88077480 x1874672517942528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828598.165801:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517942528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828623 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828598.165809:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828598.165812:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828598.165816:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517942528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828598.484780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828598.996772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828598.996808:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828599.188780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828599.188806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828599.508741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828600.020748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828600.020842:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828600.212737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828600.212767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828600.532712:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828601.044742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828601.044804:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828601.236742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828601.236787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828601.556705:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828602.068721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828602.068823:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828602.260745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828602.260758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828602.580697:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828603.092761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828603.092867:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828603.284772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828603.284790:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828603.284798:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828603.284802:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828603.284807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828603.284814:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1609 00000100:00080000:1.0:1787828603.284838:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828603.284846:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828603.284883:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517942656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828603.284907:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828603.284910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828603.284912:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828603.284915:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1609 00000100:00080000:1.0:1787828603.284918:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828603.284922:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828603.284931:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828603.284946:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517942656:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828603.284947:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517942784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828603.284969:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828603.284971:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828603.284972:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828603.284977:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1609 00000100:00080000:1.0:1787828603.284979:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828603.284982:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828603.284986:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828603.284994:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517942912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828603.285000:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828603.285002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828603.285004:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828603.285006:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1609 00000100:00080000:1.0:1787828603.285028:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828603.285032:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828603.285036:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828603.285040:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828603.285049:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517943040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828603.285058:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828603.285061:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828603.285063:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828603.285068:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1609 00000100:00080000:1.0:1787828603.285070:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828603.285073:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828603.285077:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828603.285088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517943168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828603.285095:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff8b2e88077c00 x1874672517942656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828603.285099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828603.285114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828603.285122:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828603.285138:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1609 00000100:00100000:0.0:1787828603.285140:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517942656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828603.285142:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828603.285155:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828603.285165:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828603.285166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828603.285179:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828603.285184:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517942656:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828603.285190:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517943296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828603.285197:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828603.285200:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828603.285203:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828603.285205:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1609 00000100:00080000:1.0:1787828603.285209:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828603.285211:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828603.285215:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517942784:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787828603.285219:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828603.285236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517943424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828603.285247:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828603.285264:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff8b2e88077480 x1874672517942784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828603.285273:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517942784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828603.285281:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828603.285285:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828603.285288:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517942784:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828603.285296:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517942912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828603.285323:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828603.285334:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff8b2e88075a40 x1874672517942912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828603.285346:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517942912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828603.285352:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828603.285354:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828603.285357:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517942912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828603.285361:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517943040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828603.285375:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828603.285382:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff8b2e88075680 x1874672517943040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828603.285390:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517943040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828603.285395:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828603.285397:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828603.285399:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517943040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828603.285403:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517943168:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828603.285414:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828603.285420:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff8b2e88074f00 x1874672517943168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828603.285433:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517943168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828603.285438:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828603.285440:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828603.285442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517943168:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828603.285446:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517943296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828603.285456:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828603.285462:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff8b2e88076940 x1874672517943296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828603.285469:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517943296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828603.285474:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828603.285476:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828603.285478:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517943296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828603.285491:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517943424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828603.285501:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828603.285507:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828603/real 1787828603] req@ffff8b2e88074780 x1874672517943424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828603.285515:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517943424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828628 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828603.285524:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828603.285527:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828603.285529:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517943424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828603.604715:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828604.116737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828604.116860:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828604.308722:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828604.308728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828604.628702:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828605.140743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828605.140880:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828605.332739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828605.332739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828605.652704:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828606.164713:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828606.164814:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828606.356756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828606.356757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828606.676740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828607.188792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828607.188868:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828607.380746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828607.380746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828607.700717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828608.212736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828608.212861:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828608.404721:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828608.404776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828608.404811:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828608.404817:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828608.404822:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828608.404831:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1614 00000100:00080000:3.0:1787828608.404844:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828608.404852:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828608.404892:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517943552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828608.404917:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828608.404921:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828608.404923:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828608.404928:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1614 00000100:00080000:3.0:1787828608.404932:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828608.404935:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828608.404944:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828608.404947:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517943552:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828608.404958:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672517943680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828608.404971:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828608.404974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828608.404975:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828608.404978:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1614 00000100:00080000:3.0:1787828608.404982:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828608.404985:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828608.404989:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828608.405001:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672517943808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828608.405010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828608.405014:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828608.405015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828608.405019:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828608.405023:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1614 00000100:00080000:3.0:1787828608.405026:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828608.405030:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828608.405036:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828608.405047:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672517943936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828608.405055:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828608.405057:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828608.405061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828608.405062:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff8b2e8803a580 x1874672517943552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828608.405066:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1614 00000100:00080000:3.0:1787828608.405073:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828608.405079:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828608.405083:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828608.405085:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517943552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828608.405094:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672517944064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828608.405096:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.405105:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787828608.405108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828608.405112:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517943552:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828608.405114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828608.405119:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828608.405123:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1614 00000100:00080000:3.0:1787828608.405126:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828608.405129:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828608.405134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828608.405137:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517943680:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828608.405150:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517944192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828608.405160:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828608.405163:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828608.405165:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828608.405170:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828608.405171:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1614 00000100:00080000:3.0:1787828608.405178:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828608.405182:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828608.405187:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828608.405195:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff8b2e8803b840 x1874672517943680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828608.405201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517944320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828608.405232:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517943680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.405240:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.405244:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828608.405246:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517943680:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828608.405251:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517943808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828608.405267:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828608.405275:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff8b2e88038f00 x1874672517943808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828608.405283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517943808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.405288:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.405290:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828608.405292:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517943808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828608.405296:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517943936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828608.405305:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828608.405311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff8b2e8803b0c0 x1874672517943936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828608.405318:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517943936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.405323:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.405325:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828608.405327:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517943936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828608.405331:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517944064:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828608.405340:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828608.405346:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff8b2e88039a40 x1874672517944064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828608.405357:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517944064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.405366:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.405369:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828608.405373:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517944064:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828608.405391:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517944192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828608.405422:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828608.405434:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff8b2e8803a580 x1874672517944192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828608.405447:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517944192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.405452:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.405455:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828608.405457:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517944192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828608.405461:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517944320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828608.405476:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828608.405483:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828608/real 1787828608] req@ffff8b2e880383c0 x1874672517944320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828608.405494:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517944320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828633 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828608.405502:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828608.405514:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828608.405518:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517944320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828608.724717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828609.236725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828609.236817:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828609.428739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828609.428739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828609.748708:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828610.260720:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828610.260829:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828610.452707:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828610.452720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828610.772717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828611.284751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828611.284820:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828611.476753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828611.476753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828611.796780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828612.308735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828612.308816:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828612.500716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828612.500734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828612.820752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828613.332765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828613.332848:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828613.524767:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828613.524780:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828613.524785:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828613.524798:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1619 00000100:00100000:2.0:1787828613.524801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828613.524810:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828613.524821:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828613.524822:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828613.524873:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517944448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828613.524902:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828613.524906:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828613.524908:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828613.524913:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1619 00000100:00080000:0.0:1787828613.524918:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828613.524923:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828613.524934:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828613.524939:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517944448:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828613.524952:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517944576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828613.524966:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828613.524970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828613.524972:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828613.524976:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1619 00000100:00080000:0.0:1787828613.524981:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828613.524985:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828613.524991:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787828613.525003:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828613.525007:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517944704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828613.525019:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828613.525023:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828613.525026:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828613.525030:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1619 00000100:00080000:0.0:1787828613.525034:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828613.525039:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828613.525044:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828613.525051:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff8b2e88041e00 x1874672517944448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828613.525059:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517944832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828613.525096:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828613.525100:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828613.525102:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828613.525104:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517944448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828613.525108:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1619 00000100:00080000:0.0:1787828613.525115:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828613.525123:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787828613.525125:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828613.525128:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828613.525132:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828613.525138:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517944448:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828613.525145:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517944960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828613.525154:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828613.525156:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828613.525157:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828613.525160:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1619 00000100:00080000:0.0:1787828613.525162:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828613.525165:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517944576:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828613.525168:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828613.525175:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828613.525186:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517945088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828613.525194:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828613.525197:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828613.525198:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828613.525201:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828613.525205:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1619 00000100:00080000:0.0:1787828613.525210:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828613.525213:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828613.525217:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828613.525223:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff8b2e88040b40 x1874672517944576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828613.525227:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517945216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828613.525262:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517944576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828613.525273:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828613.525276:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828613.525280:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517944576:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828613.525286:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517944704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828613.525302:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828613.525311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff8b2e880421c0 x1874672517944704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828613.525322:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517944704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828613.525336:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828613.525339:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828613.525342:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517944704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828613.525347:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517944832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828613.525361:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828613.525369:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff8b2e88042d00 x1874672517944832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828613.525379:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517944832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828613.525387:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828613.525390:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828613.525393:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517944832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828613.525398:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517944960:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828613.525422:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828613.525431:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff8b2e88043840 x1874672517944960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828613.525442:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517944960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828613.525450:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828613.525453:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828613.525456:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517944960:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828613.525471:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517945088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828613.525485:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828613.525493:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff8b2e88040780 x1874672517945088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828613.525504:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517945088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828613.525511:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828613.525514:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828613.525517:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517945088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828613.525521:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517945216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828613.525534:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828613.525552:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828613/real 1787828613] req@ffff8b2e88042580 x1874672517945216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828613.525563:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672517945216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828638 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828613.525571:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828613.525574:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828613.525578:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517945216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828613.844766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828614.356765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828614.356830:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828614.548796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828614.548798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828614.868720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828615.380767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828615.380835:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828615.572732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828615.572732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828615.892768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828616.404746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828616.404825:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828616.596722:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828616.596764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828616.916807:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828617.428740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828617.428832:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828617.620742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828617.620755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828617.940793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828618.452741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828618.452827:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828618.644776:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828618.644780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828618.644781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828618.644790:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828618.644794:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828618.644805:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1625 00000100:00080000:3.0:1787828618.644816:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828618.644824:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828618.644861:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672517945344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828618.644884:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828618.644887:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828618.644888:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828618.644891:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1625 00000100:00080000:3.0:1787828618.644894:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828618.644897:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828618.644906:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828618.644916:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672517945472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828618.644922:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517945344:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828618.644926:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828618.644932:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828618.644934:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828618.644939:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1625 00000100:00080000:3.0:1787828618.644942:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828618.644946:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828618.644952:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828618.644966:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672517945600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828618.644978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828618.644982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828618.644984:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828618.644996:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1625 00000800:00000100:0.0:1787828618.645003:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828618.645003:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828618.645011:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828618.645015:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828618.645026:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672517945728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828618.645033:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828618.645036:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828618.645037:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828618.645040:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1625 00000100:00080000:3.0:1787828618.645043:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828618.645046:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828618.645050:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828618.645056:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff8b2e880392c0 x1874672517945344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828618.645062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672517945856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828618.645081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828618.645087:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517945344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828618.645087:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828618.645092:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828618.645101:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1625 00000100:00080000:0.0:1787828618.645108:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828618.645109:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828618.645117:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828618.645119:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828618.645126:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517945344:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787828618.645126:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828618.645147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517945984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828618.645155:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517945472:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787828618.645167:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828618.645170:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828618.645171:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828618.645175:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1625 00000100:00080000:3.0:1787828618.645179:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828618.645183:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828618.645187:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828618.645188:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828618.645201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517946112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828618.645210:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff8b2e8803b480 x1874672517945472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.645229:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517945472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.645239:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.645242:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828618.645246:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517945472:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828618.645253:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517945600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828618.645271:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828618.645281:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff8b2e88039680 x1874672517945600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.645294:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517945600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.645303:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.645306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828618.645309:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517945600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828618.645315:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517945728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828618.645331:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828618.645341:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff8b2e88039e00 x1874672517945728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.645353:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517945728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.645362:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.645365:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828618.645369:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517945728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828618.645374:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517945856:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828618.645390:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828618.645399:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff8b2e8803a1c0 x1874672517945856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.645418:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517945856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.645428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.645431:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828618.645434:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517945856:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828618.645450:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517945984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828618.645476:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828618.645491:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff8b2e880383c0 x1874672517945984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.645504:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517945984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.645513:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.645516:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828618.645520:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517945984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828618.645526:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517946112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828618.645542:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828618.645551:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828618/real 1787828618] req@ffff8b2e8803a580 x1874672517946112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828618.645562:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517946112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828643 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828618.645577:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828618.645580:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828618.645584:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517946112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828618.964764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828619.476760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828619.476870:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828619.668756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828619.668765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828619.988725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828620.500740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828620.500796:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828620.692757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828620.692761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828621.012720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828621.524728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828621.524805:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828621.716739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828621.716759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828622.036767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787828622.548737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828622.548765:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828622.740757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828622.740757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828623.060728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828623.572764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828623.572802:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828623.764749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828623.764758:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.764767:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787828623.764770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828623.764772:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.764784:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1630 00000100:00080000:1.0:1787828623.764797:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.764804:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828623.764840:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517946240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828623.764866:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.764870:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828623.764873:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.764878:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1630 00000100:00080000:1.0:1787828623.764882:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.764886:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828623.764896:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828623.764898:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517946240:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828623.764913:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517946368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828623.764926:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.764929:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828623.764931:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.764936:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1630 00000100:00080000:1.0:1787828623.764947:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.764954:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787828623.764955:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787828623.764963:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828623.764976:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517946496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828623.764986:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.764990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828623.764992:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.764996:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1630 00000100:00100000:0.0:1787828623.765000:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff8b2e88077840 x1874672517946240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828623.765002:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.765011:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828623.765018:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828623.765021:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517946240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828623.765031:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828623.765035:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517946624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828623.765037:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828623.765044:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517946240:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828623.765055:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.765059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828623.765061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.765067:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1630 00000100:00100000:0.0:1787828623.765070:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517946368:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828623.765072:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.765079:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828623.765085:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828623.765091:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828623.765099:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517946752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828623.765104:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff8b2e880770c0 x1874672517946368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828623.765125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828623.765128:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517946368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828623.765131:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828623.765135:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828623.765145:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828623.765145:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1630 00000100:00080000:1.0:1787828623.765150:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828623.765154:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828623.765158:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828623.765163:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517946368:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787828623.765165:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828623.765178:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517946496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828623.765183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517946880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828623.765194:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828623.765197:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828623.765200:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828623.765204:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1630 00000800:00000100:0.0:1787828623.765210:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828623.765212:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828623.765219:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828623.765224:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828623.765228:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff8b2e88076580 x1874672517946496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828623.765239:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517947008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828623.765245:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517946496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828623.765267:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828623.765270:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828623.765273:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517946496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828623.765278:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517946624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828623.765294:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828623.765301:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff8b2e88076d00 x1874672517946624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828623.765308:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517946624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828623.765314:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828623.765316:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828623.765318:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517946624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828623.765328:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517946752:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828623.765337:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828623.765343:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff8b2e880761c0 x1874672517946752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828623.765351:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517946752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828623.765356:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828623.765358:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828623.765360:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517946752:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828623.765363:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517946880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828623.765373:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828623.765378:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff8b2e88075e00 x1874672517946880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828623.765386:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517946880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828623.765391:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828623.765392:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828623.765394:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517946880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828623.765397:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517947008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828623.765406:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828623.765411:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828623/real 1787828623] req@ffff8b2e88074b40 x1874672517947008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828623.765418:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517947008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828648 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828623.765428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828623.765430:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828623.765432:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517947008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828624.084745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828624.596725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828624.596796:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828624.788766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828624.788766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828625.108707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828625.620726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828625.620779:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828625.812743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828625.812743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828626.132737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828626.644759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828626.644842:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828626.836753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828626.836753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828627.156782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828627.668749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828627.668814:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828627.860737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828627.860740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828628.180810:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828628.692733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828628.692817:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828628.884741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828628.884742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828628.884749:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.884755:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828628.884759:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.884767:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1635 00000100:00080000:1.0:1787828628.884777:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.884783:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828628.884818:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517947136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828628.884841:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.884844:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828628.884845:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.884848:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1635 00000100:00080000:1.0:1787828628.884850:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.884853:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828628.884861:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828628.884874:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517947136:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828628.884874:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517947264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828628.884888:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.884890:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828628.884891:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.884893:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1635 00000100:00080000:1.0:1787828628.884896:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.884898:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828628.884901:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828628.884909:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517947392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828628.884915:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.884917:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828628.884918:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.884920:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1635 00000100:00080000:1.0:1787828628.884925:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.884927:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828628.884930:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828628.884935:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828628.884940:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517947520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828628.884948:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.884951:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828628.884952:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.884955:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1635 00000100:00080000:1.0:1787828628.884958:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.884960:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828628.884962:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828628.884970:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517947648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828628.884976:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff8b2e88074000 x1874672517947136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828628.884977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828628.884985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828628.884988:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828628.884995:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1635 00000100:00080000:1.0:1787828628.884998:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828628.885001:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828628.885004:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828628.885007:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517947136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828628.885013:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517947776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828628.885032:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828628.885033:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828628.885035:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828628.885039:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828628.885043:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828628.885044:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1635 00000100:00080000:1.0:1787828628.885050:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828628.885055:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517947136:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828628.885055:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828628.885058:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828628.885067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517947904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828628.885079:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517947264:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828628.885107:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828628.885121:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff8b2e88074b40 x1874672517947264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828628.885135:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517947264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828628.885142:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828628.885145:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828628.885147:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517947264:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828628.885152:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517947392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828628.885168:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828628.885176:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff8b2e88075e00 x1874672517947392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828628.885183:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517947392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828628.885189:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828628.885191:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828628.885194:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517947392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828628.885200:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517947520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828628.885218:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828628.885229:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff8b2e880761c0 x1874672517947520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828628.885242:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517947520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828628.885249:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828628.885252:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828628.885254:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517947520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828628.885258:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517947648:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828628.885273:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828628.885280:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff8b2e88076d00 x1874672517947648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828628.885288:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517947648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828628.885294:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828628.885296:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828628.885298:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517947648:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828628.885301:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517947776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828628.885311:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828628.885317:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff8b2e88076580 x1874672517947776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828628.885324:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517947776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828628.885329:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828628.885331:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828628.885332:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517947776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828628.885345:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517947904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828628.885356:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828628.885363:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828628/real 1787828628] req@ffff8b2e880770c0 x1874672517947904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828628.885370:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517947904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828653 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828628.885384:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828628.885386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828628.885388:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517947904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828629.204719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828629.716743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828629.716818:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828629.908720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828629.908721:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828630.228724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828630.740745:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828630.740805:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828630.932753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828630.932753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828631.252755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828631.764750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828631.764846:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828631.956760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828631.956763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828632.276789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828632.788758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828632.788822:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828632.980765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828632.980765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828633.300817:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828633.812737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828633.812781:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828634.004720:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828634.004725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828634.004726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828634.004732:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828634.004736:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828634.004745:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1640 00000100:00080000:1.0:1787828634.004754:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828634.004760:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828634.004795:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517948032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828634.004817:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828634.004820:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828634.004821:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828634.004824:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1640 00000100:00080000:1.0:1787828634.004826:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828634.004829:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828634.004837:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828634.004846:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517948160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828634.004852:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517948032:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828634.004856:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828634.004860:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828634.004862:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828634.004865:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1640 00000100:00080000:1.0:1787828634.004867:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828634.004870:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828634.004873:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828634.004882:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517948288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828634.004892:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828634.004895:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828634.004897:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828634.004901:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1640 00000100:00080000:1.0:1787828634.004905:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828634.004911:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828634.004912:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828634.004921:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828634.004934:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517948416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828634.004941:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828634.004944:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828634.004945:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828634.004949:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1640 00000100:00080000:1.0:1787828634.004952:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828634.004956:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828634/real 1787828634] req@ffff8b2e88077840 x1874672517948032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828634.004959:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828634.004968:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828634.004976:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517948032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828634.004985:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517948544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828634.004986:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828634.004996:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828634.005001:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517948032:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828634.005009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828634.005012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828634.005013:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828634.005017:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1640 00000100:00080000:1.0:1787828634.005021:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828634.005024:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828634.005028:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828634.005031:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517948160:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828634.005039:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517948672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828634.005047:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828634.005049:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828634.005051:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828634.005054:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1640 00000100:00080000:1.0:1787828634.005058:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828634.005061:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787828634.005067:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787828634.005067:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828634.005084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517948800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828634.005089:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828634/real 1787828634] req@ffff8b2e88074780 x1874672517948160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828634.005114:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517948160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828634.005125:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828634.005129:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828634.005133:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517948160:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828634.005141:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517948288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828634.005163:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828634.005176:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828634/real 1787828634] req@ffff8b2e88076940 x1874672517948288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828634.005189:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517948288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828634.005198:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828634.005202:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828634.005206:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517948288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828634.005212:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517948416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828634.005234:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828634.005247:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828634/real 1787828634] req@ffff8b2e88074f00 x1874672517948416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828634.005259:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517948416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828634.005268:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828634.005272:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828634.005275:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517948416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828634.005282:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517948544:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828634.005302:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828634.005314:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828634/real 1787828634] req@ffff8b2e88075680 x1874672517948544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828634.005327:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517948544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828634.005337:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828634.005340:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828634.005344:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517948544:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828634.005376:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517948672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828634.005396:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828634.005408:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828634/real 1787828634] req@ffff8b2e88075a40 x1874672517948672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828634.005421:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517948672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828634.005430:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828634.005434:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828634.005437:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517948672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828634.005444:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517948800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828634.005463:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828634.005475:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828634/real 1787828634] req@ffff8b2e88077480 x1874672517948800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828634.005487:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517948800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828659 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828634.005502:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828634.005506:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828634.005509:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517948800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828634.324744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828634.836744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828634.836798:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828635.028743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828635.028743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828635.348792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828635.860753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828635.860843:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828636.052816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828636.052880:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828636.372797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787828636.884838:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828636.884880:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787828637.076808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828637.076815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828637.396796:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828637.908725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828637.908815:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828638.100763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828638.100763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828638.420734:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828638.932739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828638.932796:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828639.124745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828639.124746:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828639.124747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828639.124765:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828639.124772:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828639.124781:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1645 00000100:00080000:1.0:1787828639.124801:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828639.124807:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828639.124859:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517948928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828639.124883:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828639.124886:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828639.124888:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828639.124891:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1645 00000100:00080000:1.0:1787828639.124894:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828639.124897:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828639.124906:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828639.124922:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517948928:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828639.124922:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517949056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828639.124938:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828639.124941:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828639.124942:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828639.124945:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1645 00000100:00080000:1.0:1787828639.124947:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828639.124949:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828639.124954:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828639.124963:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517949184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828639.124970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828639.124972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828639.124974:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828639.124977:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1645 00000100:00080000:1.0:1787828639.124980:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828639.124982:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828639.124986:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828639.124991:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828639.124999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517949312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828639.125008:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828639.125011:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828639.125012:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828639.125015:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1645 00000100:00080000:1.0:1787828639.125018:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828639.125020:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828639.125023:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828639.125034:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517949440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828639.125039:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828639/real 1787828639] req@ffff8b2e88077c00 x1874672517948928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828639.125043:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828639.125050:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828639.125053:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828639.125062:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1645 00000100:00080000:1.0:1787828639.125067:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828639.125069:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828639.125073:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828639.125081:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517948928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828639.125082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517949568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828639.125120:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828639.125123:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828639.125123:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828639.125125:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828639.125131:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828639.125132:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1645 00000100:00080000:1.0:1787828639.125138:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828639.125141:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517948928:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828639.125145:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828639.125151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828639.125159:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517949696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828639.125170:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517949056:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828639.125195:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828639.125206:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828639/real 1787828639] req@ffff8b2e88077480 x1874672517949056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828639.125215:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517949056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828639.125222:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828639.125225:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828639.125227:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517949056:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828639.125232:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517949184:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828639.125245:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828639.125252:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828639/real 1787828639] req@ffff8b2e88075a40 x1874672517949184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828639.125260:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517949184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828639.125266:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828639.125268:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828639.125270:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517949184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828639.125274:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517949312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828639.125285:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828639.125291:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828639/real 1787828639] req@ffff8b2e88075680 x1874672517949312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828639.125299:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517949312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828639.125304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828639.125306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828639.125308:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517949312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828639.125312:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517949440:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828639.125323:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828639.125330:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828639/real 1787828639] req@ffff8b2e88074f00 x1874672517949440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828639.125338:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517949440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828639.125344:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828639.125346:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828639.125348:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517949440:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828639.125352:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517949568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828639.125363:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828639.125369:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828639/real 1787828639] req@ffff8b2e88076940 x1874672517949568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828639.125389:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517949568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828639.125394:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828639.125396:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828639.125398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517949568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828639.125411:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517949696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828639.125422:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828639.125428:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828639/real 1787828639] req@ffff8b2e88074780 x1874672517949696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828639.125436:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517949696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828664 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828639.125447:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828639.125449:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828639.125451:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517949696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828639.444828:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828639.956730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828639.956784:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828640.148738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828640.148738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828640.468743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828640.980722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828640.980782:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828641.172711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828641.172713:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828641.492749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828642.004769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828642.004837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828642.196767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828642.196767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828642.516730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828643.028718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828643.028781:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828643.220716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828643.220725:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828643.540731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828644.052748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828644.052813:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828644.244716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828644.244769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828644.244772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828644.244781:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828644.244785:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828644.244797:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1650 00000100:00080000:2.0:1787828644.244808:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828644.244814:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828644.244853:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517949824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828644.244888:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828644.244891:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828644.244893:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828644.244896:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1650 00000100:00080000:2.0:1787828644.244900:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828644.244904:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828644.244912:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828644.244922:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517949952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828644.244929:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517949824:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787828644.244937:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828644.244941:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828644.244943:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828644.244946:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1650 00000100:00080000:2.0:1787828644.244948:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828644.244950:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828644.244955:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828644.244968:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517950080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828644.244977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828644.244980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828644.244982:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828644.244987:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1650 00000100:00080000:2.0:1787828644.244991:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828644.244997:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828644.245002:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828644.245016:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517950208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828644.245024:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828644.245029:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828644.245032:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828644.245033:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828644.245040:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1650 00000100:00080000:2.0:1787828644.245045:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828644.245048:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828644.245052:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828644.245063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517950336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828644.245069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828644.245071:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828644.245073:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828644.245075:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1650 00000100:00080000:2.0:1787828644.245078:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828644.245081:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828644.245086:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828644.245093:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517950464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828644.245103:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828644.245109:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828644.245112:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff8b2e8801e940 x1874672517949824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828644.245112:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828644.245125:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1650 00000100:00080000:2.0:1787828644.245134:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828644.245146:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828644.245154:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828644.245164:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517949824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828644.245169:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517950592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828644.245204:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828644.245211:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828644.245216:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517949824:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828644.245245:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517949952:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828644.245267:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828644.245277:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff8b2e8801e580 x1874672517949952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828644.245286:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517949952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828644.245292:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828644.245295:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828644.245297:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517949952:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828644.245303:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517950080:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828644.245316:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828644.245322:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff8b2e8801c000 x1874672517950080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828644.245330:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672517950080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828644.245336:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828644.245338:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828644.245339:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517950080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828644.245344:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517950208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828644.245354:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828644.245360:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff8b2e8801cf00 x1874672517950208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828644.245368:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672517950208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828644.245373:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828644.245375:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828644.245376:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517950208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828644.245381:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517950336:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828644.245391:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828644.245397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff8b2e8801f480 x1874672517950336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828644.245405:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517950336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828644.245411:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828644.245413:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828644.245415:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517950336:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828644.245419:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517950464:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828644.245429:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828644.245435:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff8b2e8801fc00 x1874672517950464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828644.245443:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517950464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828644.245465:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828644.245472:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828644.245475:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517950464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828644.245479:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517950592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828644.245493:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828644.245500:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828644/real 1787828644] req@ffff8b2e8801da40 x1874672517950592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828644.245508:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672517950592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828669 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828644.245534:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828644.245553:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828644.245557:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517950592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828644.564731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828645.076718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828645.076784:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828645.268756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828645.268763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828645.588775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828646.100757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828646.100836:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828646.292753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828646.292770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828646.612801:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828647.124727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828647.124832:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828647.316720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828647.316723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828647.636731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828648.148759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828648.148842:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828648.340761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828648.340767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828648.660726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828649.172741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828649.172804:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828649.364732:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828649.364733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828649.364745:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828649.364749:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828649.364760:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1655 00000100:00100000:3.0:1787828649.364767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828649.364772:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828649.364798:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828649.364831:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517950720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828649.364852:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828649.364854:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828649.364856:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828649.364859:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1655 00000100:00080000:2.0:1787828649.364861:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828649.364864:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828649.364889:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828649.364902:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517950848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828649.364916:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517950720:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787828649.364917:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828649.364927:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828649.364931:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828649.364953:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1655 00000100:00080000:2.0:1787828649.364957:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828649.364962:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828649.364968:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828649.364980:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517950976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828649.364988:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828649.364990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828649.364992:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828649.364994:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1655 00000100:00080000:2.0:1787828649.364997:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828649.365000:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828649.365003:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828649.365010:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787828649.365012:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517951104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828649.365020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828649.365022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828649.365024:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828649.365027:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1655 00000100:00080000:2.0:1787828649.365029:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828649.365031:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828649.365033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828649.365040:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517951232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828649.365046:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828649.365048:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828649.365049:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828649.365052:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1655 00000100:00080000:2.0:1787828649.365054:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828649.365057:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828649.365060:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828649.365068:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517951360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828649.365074:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff8b2e8801c780 x1874672517950720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828649.365076:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828649.365083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828649.365086:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828649.365097:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1655 00000100:00080000:2.0:1787828649.365103:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828649.365108:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828649.365113:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517950720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828649.365116:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828649.365131:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828649.365136:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517951488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828649.365141:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828649.365151:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517950720:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828649.365179:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517950848:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828649.365217:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828649.365233:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff8b2e8801ed00 x1874672517950848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828649.365248:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517950848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828649.365258:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828649.365262:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828649.365266:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517950848:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828649.365273:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517950976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828649.365295:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828649.365307:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff8b2e8801cb40 x1874672517950976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828649.365320:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517950976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828649.365328:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828649.365332:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828649.365336:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517950976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828649.365342:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517951104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828649.365362:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828649.365374:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff8b2e8801c3c0 x1874672517951104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828649.365387:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517951104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828649.365396:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828649.365400:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828649.365403:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517951104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828649.365410:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517951232:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828649.365431:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828649.365444:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff8b2e8801d680 x1874672517951232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828649.365458:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517951232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828649.365468:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828649.365472:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828649.365475:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517951232:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828649.365482:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517951360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828649.365518:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828649.365532:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff8b2e8801de00 x1874672517951360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828649.365544:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517951360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828649.365551:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828649.365555:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828649.365558:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517951360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828649.365565:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517951488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828649.365586:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828649.365597:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828649/real 1787828649] req@ffff8b2e8801f0c0 x1874672517951488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828649.365610:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517951488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828674 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828649.365626:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828649.365629:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828649.365633:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517951488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828649.684773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828650.196809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828650.196893:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828650.388746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828650.388791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828650.708795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828651.220798:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828651.220871:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828651.412703:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828651.412711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828651.732709:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828652.244745:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828652.244813:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828652.436747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828652.436747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828652.756773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828653.268750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828653.268835:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828653.460751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828653.460751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828653.780732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828654.292748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828654.292815:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828654.484721:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828654.484750:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828654.484760:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828654.484766:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828654.484770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828654.484779:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1660 00000100:00080000:3.0:1787828654.484795:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828654.484806:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828654.484855:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672517951616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828654.484886:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828654.484891:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828654.484894:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828654.484900:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1660 00000100:00080000:3.0:1787828654.484905:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828654.484911:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828654.484923:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517951616:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787828654.484924:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828654.484945:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672517951744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828654.484959:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828654.484964:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828654.484966:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828654.484972:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1660 00000100:00080000:3.0:1787828654.484977:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828654.484982:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828654.484989:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1787828654.484997:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828654.485006:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672517951872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828654.485020:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828654.485024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828654.485026:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828654.485031:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1660 00000100:00080000:3.0:1787828654.485036:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828654.485041:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828654.485046:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff8b2e88039a40 x1874672517951616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828654.485050:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828654.485071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672517952000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828654.485083:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828654.485087:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517951616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828654.485090:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828654.485103:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828654.485105:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828654.485111:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1660 00000100:00080000:0.0:1787828654.485115:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787828654.485119:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828654.485124:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517951616:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828654.485129:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828654.485137:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828654.485154:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517952128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828654.485157:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517951744:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787828654.485184:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828654.485187:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828654.485193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828654.485196:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828654.485202:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff8b2e8803b0c0 x1874672517951744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828654.485204:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1660 00000100:00080000:3.0:1787828654.485214:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828654.485219:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517951744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828654.485224:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828654.485233:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828654.485236:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828654.485238:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828654.485244:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517951744:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828654.485250:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517951872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828654.485259:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517952256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828654.485268:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828654.485278:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff8b2e88038f00 x1874672517951872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828654.485278:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828654.485290:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828654.485295:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828654.485296:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517951872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828654.485307:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1660 00000100:00080000:0.0:1787828654.485309:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828654.485315:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787828654.485316:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828654.485319:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517951872:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787828654.485325:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828654.485326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517952000:192.168.203.101@tcp:8:kworker.0 00000008:00100000:3.0:1787828654.485338:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828654.485345:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828654.485354:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff8b2e8803b840 x1874672517952000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828654.485358:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672517952384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828654.485372:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517952000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828654.485383:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828654.485386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828654.485390:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517952000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828654.485402:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517952128:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828654.485415:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828654.485421:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff8b2e8803a580 x1874672517952128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828654.485430:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517952128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828654.485436:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828654.485438:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828654.485439:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517952128:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828654.485443:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517952256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828654.485454:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828654.485460:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff8b2e880383c0 x1874672517952256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828654.485483:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517952256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828654.485489:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828654.485492:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828654.485494:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517952256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828654.485499:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517952384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828654.485513:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828654.485520:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828654/real 1787828654] req@ffff8b2e8803a1c0 x1874672517952384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828654.485527:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517952384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828679 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828654.485538:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828654.485540:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828654.485543:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517952384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828654.804817:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828655.316752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828655.316843:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828655.508767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828655.508778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828655.828716:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828656.340749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828656.340858:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828656.532748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828656.532763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828656.852781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828657.364732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828657.364832:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828657.556752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828657.556807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828657.876786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828658.388763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828658.388860:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828658.580708:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828658.580749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828658.900753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828659.412774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828659.412906:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828659.604734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828659.604818:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.604826:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828659.604836:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828659.604836:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828659.604856:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1666 00000100:00080000:0.0:1787828659.604869:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.604907:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828659.604953:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517952512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.605004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.605008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828659.605010:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.605014:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1666 00000100:00080000:0.0:1787828659.605019:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.605023:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828659.605034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828659.605052:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517952512:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828659.605053:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517952640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.605078:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.605083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828659.605088:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.605094:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1666 00000100:00080000:0.0:1787828659.605100:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.605106:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828659.605115:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828659.605134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517952768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.605150:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.605153:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828659.605155:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.605159:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1666 00000100:00080000:0.0:1787828659.605164:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.605168:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828659.605174:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828659.605190:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517952896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.605199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.605203:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828659.605205:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.605212:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1666 00000100:00100000:3.0:1787828659.605215:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517952640:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828659.605217:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.605227:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828659.605235:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828659.605252:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517953024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828659.605263:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.605268:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828659.605271:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828659.605278:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517952768:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787828659.605282:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1666 00000100:00080000:0.0:1787828659.605294:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.605300:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828659.605311:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828659.605315:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517952896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828659.605336:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517953152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828659.605354:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517953024:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828659.605363:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828659.605370:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828659.605373:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828659.605382:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1666 00000100:00100000:3.0:1787828659.605385:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517953152:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787828659.605392:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828659.605402:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828659.605411:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828659.605448:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517953280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828659.605537:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517953280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828659.924730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828660.436734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828660.436785:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828660.628710:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828660.628860:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828660.948799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828661.460764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828661.460834:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828661.652756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828661.652756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828661.972762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828662.484739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828662.484818:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828662.676766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828662.676817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:0.0:1787828662.677086:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:0.0:1787828662.677160:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828662.677182:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828662.677242:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828662.677254:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828662.677269:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828662.677282:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828662.677293:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787828662.677302:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828662] req@ffff8b2e88042580 x1874672517952640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787828662.677307:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787828662.677344:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672517952640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828662.677361:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828662.677368:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828662.677373:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517952640:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828662.677382:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828662] req@ffff8b2e88040780 x1874672517952768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828662.677396:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517952768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828662.677421:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828662.677427:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828662.677430:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517952768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828662.677438:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828662] req@ffff8b2e88043840 x1874672517952896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828662.677451:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517952896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828662.677460:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828662.677463:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828662.677467:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517952896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828662.677475:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828662] req@ffff8b2e880412c0 x1874672517952512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828662.677490:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672517952512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828662.677499:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828662.677503:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828662.677507:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517952512:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828662.677541:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828662] req@ffff8b2e88040b40 x1874672517953280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828662.677556:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517953280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828662.677566:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828662.677569:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828662.677572:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517953280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828662.677580:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828662] req@ffff8b2e880421c0 x1874672517953152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828662.677591:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517953152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828662.677599:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828662.677603:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828662.677607:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517953152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828662.677626:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828659/real 1787828662] req@ffff8b2e88042d00 x1874672517953024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828662.677642:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517953024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828684 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828662.677679:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828662.677683:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828662.677687:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517953024:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828662.996775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828663.508781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828663.508930:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828663.700785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828663.700785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828664.020729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828664.532740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828664.532787:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828664.724746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828664.724753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828664.724818:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.724825:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828664.724829:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.724839:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1671 00000100:00080000:1.0:1787828664.724849:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.724857:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828664.724893:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517953408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828664.724921:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.724925:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828664.724926:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.724931:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1671 00000100:00080000:1.0:1787828664.724935:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.724938:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828664.724948:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828664.724961:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517953536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828664.724972:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.724975:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828664.724977:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.724981:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1671 00000100:00080000:1.0:1787828664.724985:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.724988:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828664.724993:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828664.725005:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517953664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828664.725014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.725018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828664.725020:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.725024:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1671 00000100:00080000:1.0:1787828664.725027:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.725036:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828664.725042:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828664.725053:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517953792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828664.725062:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.725065:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828664.725067:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.725071:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1671 00000100:00080000:1.0:1787828664.725075:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.725079:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828664.725084:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828664.725095:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517953920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828664.725105:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.725108:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828664.725110:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.725113:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1671 00000100:00080000:1.0:1787828664.725117:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.725120:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828664.725125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828664.725139:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517954048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828664.725148:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828664.725152:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828664.725154:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828664.725157:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1671 00000100:00080000:1.0:1787828664.725160:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828664.725164:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828664.725168:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828664.725180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517954176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828664.725293:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517953408:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787828664.725359:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828664.725429:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff8b2e88077840 x1874672517953408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828664.725449:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517953408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828664.725461:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828664.725466:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828664.725470:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517953408:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828664.725479:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517953536:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828664.725505:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828664.725520:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff8b2e880770c0 x1874672517953536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828664.725544:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517953536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828664.725552:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828664.725675:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828664.725681:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517953536:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828664.725689:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517953664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828664.725716:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828664.725729:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff8b2e88076580 x1874672517953664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828664.725742:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517953664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828664.725751:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828664.725755:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828664.725759:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517953664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828664.725765:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517953792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828664.725787:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828664.725799:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff8b2e88076d00 x1874672517953792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828664.725811:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517953792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828664.725826:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828664.725829:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828664.725833:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517953792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828664.725838:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517953920:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828664.725858:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828664.725869:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff8b2e880761c0 x1874672517953920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828664.725882:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517953920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828664.725890:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828664.725893:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828664.726074:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517953920:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828664.726083:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517954048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828664.726109:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828664.726123:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff8b2e88075e00 x1874672517954048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828664.726135:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517954048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828664.726144:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828664.726147:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828664.726151:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517954048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828664.726157:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517954176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828664.726179:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828664.726190:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828664/real 1787828664] req@ffff8b2e88074b40 x1874672517954176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828664.726202:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517954176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828689 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828664.726211:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828664.726214:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828664.726217:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517954176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828665.044750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828665.556758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828665.556830:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828665.748743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828665.748743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828666.068784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828666.580784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828666.580868:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828666.772755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828666.772756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828667.092737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787828667.604785:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828667.604825:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828667.796794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828667.796796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828668.116744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787828668.628839:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828668.628843:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828668.820733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828668.820734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828669.140712:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828669.652822:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828669.652902:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828669.844781:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828669.844789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828669.844789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828669.844791:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828669.844800:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828669.844813:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1676 00000100:00080000:1.0:1787828669.844822:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828669.844830:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828669.844862:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517954304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828669.844884:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828669.844887:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828669.844888:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828669.844891:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1676 00000100:00080000:1.0:1787828669.844894:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828669.844896:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828669.844903:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828669.844914:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517954432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828669.844923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828669.844927:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787828669.844928:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517954304:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828669.844929:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828669.844937:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1676 00000100:00080000:1.0:1787828669.844941:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828669.844944:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828669.844949:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828669.844962:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517954560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828669.844972:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828669.844976:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828669.844978:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828669.844983:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1676 00000100:00080000:1.0:1787828669.844987:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828669.844991:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828669.844997:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828669.844999:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828669.845011:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517954688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828669.845019:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828669.845021:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828669.845023:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828669.845026:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1676 00000100:00080000:1.0:1787828669.845029:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828669.845031:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828669.845035:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828669.845041:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517954816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828669.845048:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828669.845051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828669.845054:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828669.845055:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff8b2e88074000 x1874672517954304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828669.845058:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1676 00000100:00080000:1.0:1787828669.845062:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828669.845067:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828669.845074:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828669.845084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517954944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828669.845089:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517954304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828669.845095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828669.845102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828669.845105:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828669.845112:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1676 00000100:00080000:3.0:1787828669.845113:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828669.845115:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828669.845121:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787828669.845125:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787828669.845126:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828669.845133:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517954304:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828669.845138:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517955072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828669.845162:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517954432:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828669.845189:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828669.845204:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff8b2e88074b40 x1874672517954432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828669.845218:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517954432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828669.845227:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828669.845231:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828669.845234:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517954432:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828669.845241:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517954560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828669.845263:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828669.845275:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff8b2e88075e00 x1874672517954560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828669.845294:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517954560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828669.845303:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828669.845306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828669.845309:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517954560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828669.845315:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517954688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828669.845335:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828669.845347:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff8b2e880761c0 x1874672517954688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828669.845358:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517954688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828669.845382:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828669.845385:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828669.845389:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517954688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828669.845395:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517954816:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828669.845417:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828669.845429:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff8b2e88076d00 x1874672517954816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828669.845442:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517954816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828669.845457:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828669.845460:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828669.845464:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517954816:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828669.845471:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517954944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828669.845490:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828669.845502:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff8b2e88076580 x1874672517954944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828669.845514:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517954944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828669.845522:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828669.845525:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828669.845529:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517954944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828669.845535:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517955072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828669.845554:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828669.845566:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828669/real 1787828669] req@ffff8b2e880770c0 x1874672517955072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828669.845577:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517955072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828694 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828669.845585:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828669.845589:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828669.845593:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517955072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828670.164757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828670.676772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828670.676848:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828670.868754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828670.868773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828671.188693:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828671.700753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828671.700808:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828671.892737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828671.892744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828672.212726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828672.724806:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828672.724900:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828672.916838:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828672.916839:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828673.236782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828673.748758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828673.748859:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828673.940744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828673.940759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828674.260827:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828674.772766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828674.772857:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828674.964816:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828674.964816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828674.964823:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828674.964833:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828674.964838:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828674.964852:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1681 00000100:00080000:1.0:1787828674.964864:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828674.964871:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828674.964929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517955200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828674.964975:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828674.964981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828674.964984:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828674.964991:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1681 00000100:00080000:1.0:1787828674.964997:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828674.965019:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828674.965033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828674.965050:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517955200:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828674.965073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517955328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828674.965086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828674.965089:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828674.965091:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828674.965095:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1681 00000100:00080000:1.0:1787828674.965098:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828674.965101:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828674.965106:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828674.965116:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517955456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828674.965124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828674.965126:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828674.965128:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828674.965134:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1681 00000100:00080000:1.0:1787828674.965138:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828674.965140:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828674.965143:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828674.965151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517955584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828674.965158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828674.965161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828674.965163:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828674.965176:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1681 00000100:00080000:1.0:1787828674.965179:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828674.965181:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828674.965185:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828674.965192:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828674.965197:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517955712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828674.965205:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828674.965208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828674.965210:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828674.965212:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1681 00000100:00080000:1.0:1787828674.965216:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828674.965219:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828674.965223:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828674.965249:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517955840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828674.965257:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828674.965260:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828674.965262:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828674.965265:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1681 00000100:00080000:1.0:1787828674.965269:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828674.965274:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828674.965281:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828674.965291:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517955968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828674.965307:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff8b2e88077840 x1874672517955200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828674.965389:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517955200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828674.965403:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828674.965408:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828674.965413:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517955200:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828674.965479:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517955328:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828674.965532:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828674.965581:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff8b2e88074780 x1874672517955328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828674.965610:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517955328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828674.965643:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828674.965650:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828674.965657:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517955328:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828674.965730:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517955456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828674.965761:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828674.965781:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff8b2e88076940 x1874672517955456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828674.965794:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517955456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828674.965803:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828674.965807:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828674.965810:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517955456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828674.965817:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517955584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828674.965834:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828674.965843:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff8b2e88074f00 x1874672517955584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828674.965862:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517955584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828674.965870:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828674.965873:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828674.965876:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517955584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828674.965882:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517955712:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828674.965909:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828674.965918:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff8b2e88075680 x1874672517955712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828674.965947:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517955712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828674.965955:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828674.965970:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828674.965973:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517955712:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828674.965979:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517955840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828674.966014:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828674.966024:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff8b2e88075a40 x1874672517955840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828674.966035:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517955840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828674.966050:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828674.966054:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828674.966057:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517955840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828674.966075:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517955968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828674.966091:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828674.966100:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828674/real 1787828674] req@ffff8b2e88077480 x1874672517955968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828674.966111:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517955968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828699 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828674.966136:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828674.966139:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828674.966143:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517955968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828675.284742:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828675.796758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828675.796835:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828675.988745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828675.988751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828676.308757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828676.820717:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828676.820779:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828677.012749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828677.012749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828677.332728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828677.844753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828677.844904:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828678.036739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828678.036767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828678.356761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828678.868724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828678.868773:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828679.060804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828679.060812:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828679.380746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828679.892791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828679.892878:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828680.084751:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828680.084752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828680.084752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828680.084762:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828680.084766:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828680.084776:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1686 00000100:00080000:1.0:1787828680.084789:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828680.084807:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828680.084851:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517956096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828680.084886:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828680.084894:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828680.084895:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828680.084900:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1686 00000100:00080000:1.0:1787828680.084903:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828680.084907:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828680.084917:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828680.084922:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517956096:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828680.084932:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517956224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828680.084957:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828680.084960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828680.084962:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828680.084965:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1686 00000100:00080000:1.0:1787828680.084967:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828680.084969:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828680.084973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828680.084981:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517956352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828680.084987:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828680.084990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828680.084991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828680.084993:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1686 00000100:00080000:1.0:1787828680.084997:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828680.084999:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828680.085002:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828680.085010:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517956480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828680.085015:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828680.085019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828680.085021:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828680.085021:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828680.085028:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1686 00000100:00080000:1.0:1787828680.085031:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828680.085034:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828680.085037:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828680.085044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517956608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828680.085050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828680.085052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828680.085053:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828680.085056:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1686 00000100:00080000:1.0:1787828680.085059:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828680.085062:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828680.085066:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828680.085075:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828680/real 1787828680] req@ffff8b2e88077c00 x1874672517956096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828680.085076:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517956736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828680.085109:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828680.085112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828680.085113:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828680.085118:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1686 00000100:00100000:3.0:1787828680.085119:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517956096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828680.085132:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828680.085139:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828680.085146:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828680.085150:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828680.085156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517956864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828680.085156:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828680.085164:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517956096:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828680.085193:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517956224:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828680.085213:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828680.085222:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828680/real 1787828680] req@ffff8b2e88077480 x1874672517956224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828680.085230:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517956224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828680.085237:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828680.085240:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828680.085242:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517956224:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828680.085246:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517956352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828680.085260:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828680.085267:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828680/real 1787828680] req@ffff8b2e88075a40 x1874672517956352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828680.085274:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517956352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828680.085280:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828680.085281:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828680.085283:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517956352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828680.085287:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517956480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828680.085342:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828680.085354:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828680/real 1787828680] req@ffff8b2e88075680 x1874672517956480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828680.085367:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517956480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828680.085376:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828680.085380:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828680.085383:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517956480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828680.085390:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517956608:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828680.085410:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828680.085418:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828680/real 1787828680] req@ffff8b2e88074f00 x1874672517956608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828680.085432:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517956608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828680.085439:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828680.085441:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828680.085443:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517956608:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828680.085446:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517956736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828680.085457:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828680.085463:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828680/real 1787828680] req@ffff8b2e88076940 x1874672517956736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828680.085471:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517956736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828680.085476:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828680.085478:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828680.085480:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517956736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828680.085483:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517956864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828680.085493:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828680.085499:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828680/real 1787828680] req@ffff8b2e88074780 x1874672517956864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828680.085506:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517956864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828705 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828680.085517:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828680.085519:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828680.085521:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517956864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828680.404758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828680.916785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828680.916875:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828681.108820:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828681.108820:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828681.428806:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828681.940727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828681.940773:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828682.132739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828682.132744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828682.452745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828682.964752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828682.964805:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828683.156733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828683.156734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828683.476789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828683.988804:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828683.988821:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828684.180765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828684.180775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828684.500718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828685.012758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828685.012854:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828685.204770:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828685.204771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828685.204784:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828685.204790:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828685.204794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828685.204804:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1691 00000100:00080000:1.0:1787828685.204815:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828685.204827:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828685.204866:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517956992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828685.204893:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828685.204897:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828685.204898:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828685.204902:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1691 00000100:00080000:1.0:1787828685.204906:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828685.204910:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828685.204919:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828685.204931:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517957120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828685.204939:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517956992:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828685.204947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828685.204951:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828685.204953:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828685.204956:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1691 00000100:00080000:1.0:1787828685.204959:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828685.204963:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828685.204966:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828685.204976:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517957248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828685.204985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828685.204988:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828685.204990:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828685.205000:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1691 00000100:00080000:1.0:1787828685.205003:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828685.205005:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828685.205009:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828685.205016:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517957376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828685.205025:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828685.205033:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828685.205035:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828685.205042:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828685.205046:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1691 00000100:00080000:1.0:1787828685.205050:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828685.205055:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828685.205061:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828685.205073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517957504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828685.205081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828685.205083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828685.205086:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828685.205089:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1691 00000100:00080000:1.0:1787828685.205091:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828685.205100:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828685.205101:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828685/real 1787828685] req@ffff8b2e88077840 x1874672517956992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828685.205106:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828685.205122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517957632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828685.205130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828685.205136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787828685.205136:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517956992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828685.205144:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828685.205158:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1691 00000100:00080000:1.0:1787828685.205170:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828685.205174:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828685.205176:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787828685.205185:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787828685.205186:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828685.205199:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517956992:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828685.205205:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517957760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828685.205233:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517957120:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828685.205273:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828685.205287:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828685/real 1787828685] req@ffff8b2e880770c0 x1874672517957120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828685.205307:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517957120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828685.205314:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828685.205317:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828685.205319:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517957120:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828685.205324:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517957248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828685.205340:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828685.205347:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828685/real 1787828685] req@ffff8b2e88076580 x1874672517957248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828685.205355:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517957248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828685.205361:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828685.205363:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828685.205365:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517957248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828685.205369:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517957376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828685.205379:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828685.205385:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828685/real 1787828685] req@ffff8b2e88076d00 x1874672517957376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828685.205397:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517957376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828685.205403:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828685.205405:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828685.205406:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517957376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828685.205411:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517957504:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828685.205421:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828685.205427:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828685/real 1787828685] req@ffff8b2e880761c0 x1874672517957504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828685.205435:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517957504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828685.205440:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828685.205442:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828685.205444:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517957504:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828685.205448:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517957632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828685.205459:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828685.205465:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828685/real 1787828685] req@ffff8b2e88075e00 x1874672517957632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828685.205473:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517957632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828685.205482:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828685.205484:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828685.205486:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517957632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828685.205490:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517957760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828685.205500:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828685.205506:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828685/real 1787828685] req@ffff8b2e88074b40 x1874672517957760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828685.205512:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517957760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828710 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828685.205518:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828685.205521:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828685.205523:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517957760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828685.524834:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828686.036770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828686.036838:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828686.228763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828686.228788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828686.548765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828687.060737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828687.060836:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828687.252855:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828687.252855:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828687.572791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828688.084767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828688.084924:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828688.276723:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828688.276768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828688.596729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828689.108817:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828689.108933:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828689.300790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828689.300810:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828689.620775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828690.132744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828690.132888:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828690.324717:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828690.324769:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828690.324777:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828690.324782:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828690.324792:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1696 00000100:00080000:1.0:1787828690.324804:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828690.324805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828690.324814:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828690.324853:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517957888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828690.324878:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828690.324882:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828690.324884:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828690.324889:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1696 00000100:00080000:1.0:1787828690.324892:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828690.324897:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828690.324905:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517957888:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787828690.324908:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828690.324928:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517958016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828690.324946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828690.324949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828690.324951:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828690.324955:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1696 00000100:00080000:1.0:1787828690.324959:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828690.324964:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787828690.324965:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787828690.324973:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828690.324985:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517958144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828690.324993:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828690.324997:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828690.324999:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828690.325003:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1696 00000100:00080000:1.0:1787828690.325007:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828690.325011:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828690.325017:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828690.325024:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff8b2e88074000 x1874672517957888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828690.325030:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517958272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828690.325046:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828690.325048:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517957888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828690.325051:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828690.325055:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828690.325063:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828690.325064:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1696 00000100:00080000:1.0:1787828690.325070:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828690.325070:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828690.325076:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517957888:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828690.325079:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828690.325085:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828690.325096:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517958400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828690.325100:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517958016:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828690.325116:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828690.325120:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828690.325123:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828690.325128:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828690.325132:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1696 00000100:00080000:1.0:1787828690.325138:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828690.325143:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff8b2e88074b40 x1874672517958016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828690.325144:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828690.325153:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828690.325158:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517958016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828690.325167:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828690.325169:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517958528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828690.325172:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828690.325179:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517958016:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828690.325191:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517958144:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787828690.325192:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828690.325201:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828690.325203:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828690.325208:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1696 00000100:00080000:1.0:1787828690.325213:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828690.325214:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828690.325235:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828690.325241:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828690.325248:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff8b2e88075e00 x1874672517958144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828690.325257:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517958656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828690.325260:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517958144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828690.325307:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828690.325311:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828690.325315:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517958144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828690.325320:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517958272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828690.325336:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828690.325345:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff8b2e880761c0 x1874672517958272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828690.325354:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517958272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828690.325363:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828690.325366:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828690.325370:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517958272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828690.325386:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517958400:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828690.325407:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828690.325419:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff8b2e88076d00 x1874672517958400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828690.325431:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517958400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828690.325440:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828690.325443:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828690.325446:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517958400:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828690.325452:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517958528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828690.325470:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828690.325480:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff8b2e88076580 x1874672517958528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828690.325491:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517958528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828690.325499:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828690.325502:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828690.325505:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517958528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828690.325510:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517958656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828690.325525:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828690.325533:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828690/real 1787828690] req@ffff8b2e880770c0 x1874672517958656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828690.325545:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517958656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828715 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828690.325552:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828690.325555:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828690.325558:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517958656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828690.644758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828691.156715:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828691.156804:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828691.348730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828691.348730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828691.668790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828692.180754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828692.180819:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828692.372766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828692.372766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828692.692706:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828693.204773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828693.204842:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828693.396736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828693.396736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828693.716760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828694.228726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828694.228788:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828694.420777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828694.420779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828694.740718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828695.252740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828695.252812:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828695.444722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828695.444731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828695.444737:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.444745:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828695.444750:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.444759:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1701 00000100:00080000:0.0:1787828695.444768:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.444775:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828695.444833:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517958784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828695.444873:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.444877:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828695.444879:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.444884:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1701 00000100:00080000:0.0:1787828695.444888:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.444894:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828695.444903:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828695.444911:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517958784:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828695.444917:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517958912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828695.444937:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.444940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828695.444942:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.444945:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1701 00000100:00080000:0.0:1787828695.444948:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.444950:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828695.444954:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828695.444963:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517959040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828695.444969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.444971:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828695.444972:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.444975:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1701 00000100:00080000:0.0:1787828695.444978:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.444980:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828695.444982:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828695.444987:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828695.444992:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517959168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828695.444999:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.445001:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828695.445003:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.445006:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1701 00000100:00080000:0.0:1787828695.445010:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.445012:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828695.445015:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828695.445023:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517959296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828695.445030:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828695.445034:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787828695.445034:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff8b2e88041e00 x1874672517958784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828695.445036:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.445044:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1701 00000100:00080000:0.0:1787828695.445049:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.445054:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828695.445057:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828695.445061:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517958784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828695.445068:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517959424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828695.445072:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828695.445079:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828695.445080:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787828695.445086:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787828695.445089:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517958784:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828695.445090:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828695.445110:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1701 00000100:00080000:0.0:1787828695.445112:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828695.445115:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828695.445117:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828695.445123:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517959552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828695.445128:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517958912:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828695.445152:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828695.445163:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff8b2e88041a40 x1874672517958912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828695.445172:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517958912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828695.445178:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828695.445194:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828695.445197:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517958912:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828695.445202:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517959040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828695.445215:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828695.445223:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff8b2e88040f00 x1874672517959040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828695.445231:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517959040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828695.445242:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828695.445245:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828695.445247:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517959040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828695.445251:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517959168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828695.445262:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828695.445268:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff8b2e88042940 x1874672517959168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828695.445276:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517959168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828695.445281:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828695.445284:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828695.445286:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517959168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828695.445289:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517959296:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828695.445298:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828695.445305:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff8b2e88043480 x1874672517959296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828695.445312:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517959296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828695.445318:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828695.445326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828695.445328:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517959296:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828695.445331:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517959424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828695.445342:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828695.445349:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff8b2e880430c0 x1874672517959424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828695.445356:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517959424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828695.445361:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828695.445364:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828695.445365:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517959424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828695.445377:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517959552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828695.445388:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828695.445393:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828695/real 1787828695] req@ffff8b2e88040000 x1874672517959552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828695.445401:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517959552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828720 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828695.445406:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828695.445408:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828695.445411:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517959552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828695.764795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828696.276733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828696.276823:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828696.468782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828696.468788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828696.788826:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828697.300799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828697.300872:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828697.492714:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828697.492718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828697.812771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828698.324757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828698.324836:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828698.516753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828698.516753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828698.836707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828699.348739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828699.348865:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828699.540747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828699.540748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828699.860740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828700.372767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828700.372803:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828700.564803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828700.564803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828700.564863:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828700.564876:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828700.564884:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828700.564899:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1706 00000100:00080000:0.0:1787828700.564912:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828700.564921:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828700.564978:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517959680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828700.565022:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828700.565030:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828700.565033:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828700.565041:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1706 00000100:00080000:0.0:1787828700.565047:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828700.565054:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828700.565068:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828700.565077:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517959680:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828700.565094:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517959808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828700.565111:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828700.565116:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828700.565119:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828700.565124:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1706 00000100:00080000:0.0:1787828700.565130:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828700.565134:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828700.565141:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828700.565185:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517959936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828700.565197:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828700.565222:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828700.565227:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828700.565230:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828700.565236:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1706 00000100:00080000:0.0:1787828700.565242:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828700.565257:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828700.565265:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828700.565267:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828700] req@ffff8b2e880403c0 x1874672517959680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828700.565286:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517960064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828700.565299:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672517959680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828700.565302:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828700.565333:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828700.565333:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828700.565335:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828700.565339:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787828700.565342:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1706 00000100:00100000:3.0:1787828700.565345:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517959680:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828700.565347:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828700.565353:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828700.565357:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828700.565365:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517960192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828700.565371:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828700.565374:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828700.565375:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828700.565378:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517959808:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828700.565382:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1706 00000100:00080000:0.0:1787828700.565387:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828700.565401:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828700.565404:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828700.565413:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517960320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828700.565418:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828700.565429:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828700.565433:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787828700.565434:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828700] req@ffff8b2e88040000 x1874672517959808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828700.565435:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828700.565442:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1706 00000100:00080000:0.0:1787828700.565448:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828700.565453:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828700.565456:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828700.565457:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517959808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828700.565468:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517960448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828700.565468:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828700.565474:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828700.565481:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517959808:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828700.565491:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517959936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828700.565506:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828700.565514:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828700] req@ffff8b2e880430c0 x1874672517959936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828700.565522:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517959936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828700.565527:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828700.565529:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828700.565531:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517959936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828700.565534:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517960064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828700.565543:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828700.565549:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828700] req@ffff8b2e88043480 x1874672517960064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828700.565556:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517960064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828700.565561:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828700.565563:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828700.565565:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517960064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828700.565569:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517960192:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828700.565578:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828700.565584:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828700] req@ffff8b2e88042940 x1874672517960192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828700.565590:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517960192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828700.565596:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828700.565598:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828700.565600:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517960192:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828700.565611:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517960320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828700.565621:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828700.565627:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828700] req@ffff8b2e88040f00 x1874672517960320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828700.565634:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517960320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828700.565639:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828700.565640:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828700.565643:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517960320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828700.565672:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517960448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828700.565693:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828700.565706:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828700/real 1787828700] req@ffff8b2e88041a40 x1874672517960448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828700.565715:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517960448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828725 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828700.565721:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828700.565723:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828700.565725:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517960448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828700.884757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828701.396763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828701.396863:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828701.588812:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828701.588812:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828701.908743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828702.420786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828702.420858:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828702.612737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828702.612751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828702.932738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828703.444766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828703.444871:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828703.636801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828703.636805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828703.956752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828704.468749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828704.468841:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828704.660776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828704.660776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828704.980758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828705.492746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828705.492831:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828705.684720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828705.684731:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828705.684738:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828705.684743:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828705.684753:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1712 00000100:00080000:0.0:1787828705.684765:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828705.684773:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828705.684827:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517960576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828705.684850:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828705.684856:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828705.684858:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828705.684862:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1712 00000100:00080000:0.0:1787828705.684865:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828705.684867:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828705.684876:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517960576:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787828705.684877:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828705.684891:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517960704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828705.684901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828705.684904:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828705.684905:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828705.684909:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1712 00000100:00080000:0.0:1787828705.684911:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828705.684913:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828705.684919:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828705.684929:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828705.684931:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517960832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828705.684949:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828705.684952:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828705.684954:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828705.684956:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1712 00000100:00080000:0.0:1787828705.684960:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828705.684964:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828705.684966:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828705.684973:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828705.684982:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517960960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828705.684990:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828705.684993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828705.684994:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828705.684997:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1712 00000100:00080000:0.0:1787828705.684999:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828705.685001:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828705.685005:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828705.685019:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517961088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828705.685019:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff8b2e88041e00 x1874672517960576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828705.685042:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828705.685046:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828705.685048:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828705.685051:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1712 00000100:00080000:0.0:1787828705.685054:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828705.685056:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517960576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828705.685059:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828705.685065:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828705.685076:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517961216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828705.685082:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828705.685084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828705.685088:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828705.685090:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787828705.685091:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828705.685097:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1712 00000100:00100000:3.0:1787828705.685098:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517960576:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828705.685099:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828705.685105:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828705.685109:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828705.685129:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517961344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828705.685141:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517960704:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828705.685161:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828705.685171:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff8b2e88042d00 x1874672517960704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828705.685180:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517960704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828705.685186:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828705.685189:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828705.685191:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517960704:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828705.685196:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517960832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828705.685207:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828705.685213:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff8b2e880421c0 x1874672517960832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828705.685221:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517960832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828705.685227:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828705.685228:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828705.685230:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517960832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828705.685234:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517960960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828705.685244:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828705.685250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff8b2e88040b40 x1874672517960960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828705.685258:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517960960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828705.685263:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828705.685265:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828705.685267:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517960960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828705.685270:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517961088:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828705.685279:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828705.685285:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff8b2e880412c0 x1874672517961088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828705.685292:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672517961088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828705.685298:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828705.685300:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828705.685301:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517961088:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828705.685305:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517961216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828705.685314:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828705.685320:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff8b2e88043840 x1874672517961216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828705.685328:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517961216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828705.685333:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828705.685335:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828705.685336:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517961216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828705.685340:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517961344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828705.685349:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828705.685366:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828705/real 1787828705] req@ffff8b2e88040780 x1874672517961344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828705.685373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517961344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828730 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828705.685379:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828705.685380:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828705.685382:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517961344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828706.004765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828706.516756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828706.516803:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828706.708785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828706.708803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828707.028760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828707.540802:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828707.540893:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828707.732714:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828707.732719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828708.052752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828708.564757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828708.564836:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828708.756743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828708.756743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828709.076819:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828709.588771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828709.588833:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828709.780747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828709.780768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828710.100735:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828710.612782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828710.612791:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828710.804739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828710.804739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828710.804812:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828710.804821:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828710.804827:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.804838:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1717 00000100:00080000:0.0:1787828710.804853:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.804861:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828710.804930:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517961472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828710.804985:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828710.804990:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828710.804993:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.804999:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1717 00000100:00080000:0.0:1787828710.805007:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.805015:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828710.805029:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828710.805030:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517961472:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828710.805059:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517961600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828710.805120:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828710.805126:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828710.805129:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828710.805136:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828710.805141:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1717 00000100:00080000:0.0:1787828710.805149:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.805173:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828710.805182:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828710.805204:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517961728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828710.805205:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff8b2e88042580 x1874672517961472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828710.805266:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828710.805273:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672517961472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828710.805274:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828710.805280:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.805300:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1717 00000100:00080000:0.0:1787828710.805318:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.805327:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787828710.805327:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828710.805341:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:0.0:1787828710.805342:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828710.805357:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517961472:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828710.805365:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517961856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828710.805385:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828710.805389:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828710.805391:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.805398:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1717 00000100:00080000:0.0:1787828710.805402:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828710.805406:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517961600:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828710.805410:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828710.805419:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828710.805434:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517961984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828710.805446:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828710.805446:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828710.805455:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828710.805458:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.805463:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1717 00000100:00100000:3.0:1787828710.805469:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff8b2e88040780 x1874672517961600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828710.805470:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.805486:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828710.805500:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828710.805509:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517961600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828710.805520:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517962112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828710.805523:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828710.805540:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828710.805557:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517961600:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828710.805559:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828710.805576:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828710.805579:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828710.805592:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1717 00000100:00100000:3.0:1787828710.805592:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517961728:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787828710.805603:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828710.805613:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828710.805621:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828710.805621:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828710.805648:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff8b2e88043840 x1874672517961728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828710.805659:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517962240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828710.805671:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517961728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828710.805689:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828710.805693:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828710.805695:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517961728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828710.805705:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517961856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828710.805769:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828710.805779:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff8b2e880412c0 x1874672517961856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828710.805788:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672517961856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828710.805794:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828710.805797:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828710.805799:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517961856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828710.805814:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517961984:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828710.805827:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828710.805834:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff8b2e88040b40 x1874672517961984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828710.805843:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517961984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828710.805849:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828710.805851:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828710.805854:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517961984:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828710.805858:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517962112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828710.805870:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828710.805876:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff8b2e880421c0 x1874672517962112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828710.805893:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517962112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828710.805902:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828710.805905:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828710.805907:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517962112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828710.805912:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517962240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828710.805925:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828710.805932:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828710/real 1787828710] req@ffff8b2e88042d00 x1874672517962240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828710.805941:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517962240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828735 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828710.805946:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828710.805948:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828710.805950:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517962240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828711.124794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828711.636756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828711.636860:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828711.828707:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828711.828715:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828712.148735:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828712.660760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828712.660827:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828712.852739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828712.852739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828713.172778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828713.684746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828713.684777:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828713.876736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828713.876762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828714.196705:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828714.708763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828714.708811:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828714.900722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828714.900722:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828715.220769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828715.732777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828715.732856:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828715.924751:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828715.924752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828715.924752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828715.924761:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828715.924767:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.924778:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1722 00000100:00080000:0.0:1787828715.924791:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.924800:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828715.924847:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517962368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828715.924868:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828715.924871:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828715.924872:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.924875:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1722 00000100:00080000:0.0:1787828715.924878:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.924881:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828715.924891:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828715.924900:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517962496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828715.924910:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517962368:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828715.924921:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828715.924926:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828715.924928:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.924934:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1722 00000100:00080000:0.0:1787828715.924939:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.924944:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828715.924950:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828715.924964:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517962624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828715.924971:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828715.924974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828715.924976:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.924978:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1722 00000100:00080000:0.0:1787828715.924982:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.924984:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828715.924988:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828715.924998:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517962752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828715.925006:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828715.925008:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828715.925012:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828715.925013:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.925017:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1722 00000100:00080000:0.0:1787828715.925020:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.925023:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828715.925026:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828715.925033:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517962880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828715.925053:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828715.925056:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828715.925057:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.925061:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1722 00000100:00080000:0.0:1787828715.925064:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.925068:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828715.925072:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828715.925079:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff8b2e88041e00 x1874672517962368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828715.925083:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517963008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828715.925112:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828715.925116:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828715.925117:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828715.925120:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1722 00000100:00080000:0.0:1787828715.925122:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828715.925124:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828715.925129:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828715.925129:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517962368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828715.925140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517963136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828715.925150:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828715.925157:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828715.925164:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517962368:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828715.925191:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517962496:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828715.925221:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828715.925237:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff8b2e88041a40 x1874672517962496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828715.925248:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517962496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828715.925255:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828715.925257:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828715.925260:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517962496:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828715.925265:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517962624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828715.925282:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828715.925290:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff8b2e88040f00 x1874672517962624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828715.925298:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517962624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828715.925303:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828715.925306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828715.925309:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517962624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828715.925314:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517962752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828715.925344:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828715.925357:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff8b2e88042940 x1874672517962752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828715.925370:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517962752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828715.925377:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828715.925379:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828715.925382:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517962752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828715.925386:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517962880:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828715.925402:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828715.925410:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff8b2e88043480 x1874672517962880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828715.925418:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517962880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828715.925424:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828715.925426:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828715.925428:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517962880:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828715.925431:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517963008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828715.925441:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828715.925447:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff8b2e880430c0 x1874672517963008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828715.925454:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517963008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828715.925459:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828715.925461:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828715.925463:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517963008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828715.925466:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517963136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828715.925477:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828715.925482:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828715/real 1787828715] req@ffff8b2e88040000 x1874672517963136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828715.925489:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517963136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828740 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828715.925495:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828715.925497:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828715.925499:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517963136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828716.244719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828716.756745:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828716.756793:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828716.948724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828716.948728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828717.268768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828717.780718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828717.780792:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828717.972767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828717.972774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828718.292768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828718.804754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828718.804816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828718.996773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828718.996773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828719.316775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828719.828760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828719.828807:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828720.020706:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828720.020724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828720.340745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828720.852732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828720.852804:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828721.044756:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828721.044764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828721.044764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828721.044767:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828721.044773:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828721.044783:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1727 00000100:00080000:0.0:1787828721.044793:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828721.044799:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828721.044839:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517963264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828721.044860:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828721.044862:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828721.044864:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828721.044866:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1727 00000100:00080000:0.0:1787828721.044869:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828721.044871:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828721.044880:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828721.044890:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517963392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828721.044895:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517963264:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828721.044906:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828721.044909:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828721.044910:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828721.044913:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1727 00000100:00080000:0.0:1787828721.044916:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828721.044918:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828721.044921:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828721.044930:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517963520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828721.044937:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828721.044939:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828721.044940:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828721.044943:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1727 00000100:00080000:0.0:1787828721.044946:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828721.044948:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828721.044951:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828721.044956:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828721.044960:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517963648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828721.044967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828721.044969:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828721.044971:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828721.044974:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1727 00000100:00080000:0.0:1787828721.044976:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828721.044979:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828721.044984:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828721.044996:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517963776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828721.045019:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828721/real 1787828721] req@ffff8b2e880403c0 x1874672517963264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828721.045020:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828721.045031:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828721.045035:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828721.045042:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1727 00000100:00100000:3.0:1787828721.045045:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672517963264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828721.045047:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828721.045056:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787828721.045058:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787828721.045063:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828721.045064:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828721.045071:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517963264:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828721.045079:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517963904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828721.045092:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517963392:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828721.045093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828721.045102:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828721.045103:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828721.045106:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1727 00000100:00080000:0.0:1787828721.045109:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828721.045112:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828721.045116:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828721.045123:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828721.045127:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517964032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828721.045137:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828721/real 1787828721] req@ffff8b2e88040000 x1874672517963392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828721.045149:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517963392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828721.045155:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828721.045158:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828721.045160:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517963392:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828721.045165:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517963520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828721.045177:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828721.045184:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828721/real 1787828721] req@ffff8b2e880430c0 x1874672517963520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828721.045191:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517963520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828721.045197:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828721.045199:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828721.045201:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517963520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828721.045204:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517963648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828721.045213:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828721.045219:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828721/real 1787828721] req@ffff8b2e88043480 x1874672517963648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828721.045230:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517963648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828721.045235:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828721.045237:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828721.045239:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517963648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828721.045242:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517963776:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828721.045251:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828721.045256:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828721/real 1787828721] req@ffff8b2e88042940 x1874672517963776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828721.045263:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517963776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828721.045269:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828721.045270:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828721.045272:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517963776:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828721.045283:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517963904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828721.045292:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828721.045298:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828721/real 1787828721] req@ffff8b2e88040f00 x1874672517963904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828721.045305:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517963904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828721.045314:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828721.045316:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828721.045318:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517963904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828721.045321:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517964032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828721.045330:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828721.045336:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828721/real 1787828721] req@ffff8b2e88041a40 x1874672517964032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828721.045342:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517964032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828746 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828721.045348:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828721.045349:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828721.045351:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517964032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828721.364768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828721.876755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828721.876835:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828722.068764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828722.068790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828722.388705:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828722.900728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828722.900809:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828723.092779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828723.092779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828723.412780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828723.924789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828723.924884:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828724.116756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828724.116756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828724.436704:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828724.948739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828724.948861:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828725.140747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828725.140747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828725.460731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828725.972775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828725.972837:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828726.164761:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828726.164771:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828726.164774:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828726.164781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828726.164784:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1732 00000100:00100000:3.0:1787828726.164793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828726.164796:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828726.164804:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828726.164856:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517964160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828726.164895:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828726.164908:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828726.164909:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828726.164913:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1732 00000100:00080000:0.0:1787828726.164916:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828726.164920:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828726.164930:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828726.164961:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517964288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828726.164961:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517964160:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828726.164996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828726.164999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828726.165001:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828726.165005:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1732 00000100:00080000:0.0:1787828726.165007:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828726.165010:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828726.165015:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828726.165025:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517964416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828726.165031:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828726.165033:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828726.165035:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828726.165038:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1732 00000100:00080000:0.0:1787828726.165040:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828726.165042:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828726.165045:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828726.165051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517964544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828726.165058:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828726.165060:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828726.165061:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828726.165064:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1732 00000100:00080000:0.0:1787828726.165066:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828726.165069:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828726.165073:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828726.165080:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517964672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828726.165087:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828726.165088:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517964288:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828726.165093:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828726.165097:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828726.165102:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1732 00000100:00080000:0.0:1787828726.165105:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828726.165107:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828726.165110:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828726.165119:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517964800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828726.165126:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828726.165128:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828726.165130:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828726.165132:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517964416:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787828726.165135:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1732 00000100:00080000:0.0:1787828726.165143:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828726.165145:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828726.165149:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828726.165158:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517964928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828726.165166:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517964544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828726.165193:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517964672:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828726.165212:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517964800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828726.165229:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517964928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828726.484705:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828726.996752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828726.996878:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828727.188743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828727.188861:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828727.508757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828728.020752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828728.020884:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828728.212751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828728.212841:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828728.532719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828729.044725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828729.044763:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828729.236747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828729.236990:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:1.0:1787828729.237113:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787828729.237169:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787828729.237212:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787828729.237297:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787828729.237322:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787828729.237337:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787828729.237351:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787828729.237362:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787828729.237374:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787828729.237490:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828726/real 1787828729] req@ffff8b2e880412c0 x1874672517964672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828729.237530:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672517964672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828729.237550:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828729.237558:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828729.237565:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517964672:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828729.237578:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828726/real 1787828729] req@ffff8b2e88043840 x1874672517964800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828729.237597:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517964800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828729.237608:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828729.237613:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828729.237618:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517964800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828729.237627:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828726/real 1787828729] req@ffff8b2e88040780 x1874672517964928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828729.238031:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517964928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828729.238049:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828729.238055:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828729.238060:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517964928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828729.238072:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828726/real 1787828729] req@ffff8b2e88042d00 x1874672517964288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828729.238095:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517964288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828729.238109:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828729.238114:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828729.238119:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517964288:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828729.238129:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828726/real 1787828729] req@ffff8b2e880421c0 x1874672517964416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828729.238146:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517964416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828729.238157:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828729.238161:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828729.238165:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517964416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828729.238174:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828726/real 1787828729] req@ffff8b2e88040b40 x1874672517964544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828729.238191:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517964544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828729.238201:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828729.238206:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828729.238210:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517964544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828729.238220:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828726/real 1787828729] req@ffff8b2e88041e00 x1874672517964160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828729.238237:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517964160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828751 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828729.238249:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828729.238253:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828729.238258:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517964160:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828729.556749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828730.068769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828730.068878:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828730.260799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828730.260860:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828730.580770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828731.092735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828731.092837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828731.284809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828731.284863:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828731.284890:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828731.284895:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828731.284899:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828731.284908:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1737 00000100:00080000:2.0:1787828731.284921:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828731.284927:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828731.284982:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517965056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828731.285010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828731.285014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828731.285015:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828731.285019:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1737 00000100:00080000:2.0:1787828731.285022:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828731.285025:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828731.285034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828731.285044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517965184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828731.285057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828731.285061:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828731.285062:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828731.285068:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1737 00000100:00100000:3.0:1787828731.285069:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517965056:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787828731.285072:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828731.285079:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828731.285085:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828731.285099:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517965312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828731.285115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828731.285119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828731.285121:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828731.285130:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1737 00000100:00080000:2.0:1787828731.285142:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828731.285147:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828731.285153:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828731.285162:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787828731.285169:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517965440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828731.285185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828731.285190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828731.285192:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828731.285197:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1737 00000100:00080000:2.0:1787828731.285201:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828731.285207:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828731.285212:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828731.285230:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff8b2e8801f0c0 x1874672517965056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828731.285241:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517965568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828731.285257:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828731.285262:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517965056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828731.285264:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828731.285267:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828731.285275:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1737 00000100:00080000:2.0:1787828731.285284:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828731.285287:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828731.285292:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787828731.285299:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:2.0:1787828731.285301:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828731.285307:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517965056:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828731.285319:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517965696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828731.285331:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828731.285335:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828731.285337:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828731.285342:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1737 00000100:00080000:2.0:1787828731.285346:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828731.285348:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517965184:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828731.285354:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828731.285362:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828731.285378:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517965824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828731.285388:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828731.285408:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff8b2e8801de00 x1874672517965184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828731.285424:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517965184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828731.285435:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828731.285439:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828731.285443:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517965184:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828731.285449:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517965312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828731.285477:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828731.285489:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff8b2e8801d680 x1874672517965312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828731.285504:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517965312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828731.285526:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828731.285530:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828731.285534:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517965312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828731.285549:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517965440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828731.285572:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828731.285585:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff8b2e8801c3c0 x1874672517965440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828731.285596:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517965440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828731.285605:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828731.285608:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828731.285612:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517965440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828731.285756:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517965568:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828731.285783:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828731.285797:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff8b2e8801cb40 x1874672517965568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828731.285810:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517965568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828731.285826:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828731.285830:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828731.285834:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517965568:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828731.285841:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517965696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828731.285865:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828731.285877:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff8b2e8801ed00 x1874672517965696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828731.285889:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517965696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828731.285897:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828731.285901:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828731.285904:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517965696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828731.285927:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517965824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828731.285968:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828731.285985:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828731/real 1787828731] req@ffff8b2e8801c780 x1874672517965824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828731.285999:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517965824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828756 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828731.286008:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828731.286011:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828731.286015:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517965824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828731.604748:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828732.116778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828732.116882:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828732.308795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828732.308897:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828732.628710:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828733.140731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828733.140821:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828733.332769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828733.332769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828733.652697:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828734.164779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828734.164849:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828734.356779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828734.356828:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828734.676720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828735.188736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828735.188812:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828735.380768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828735.380769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828735.700712:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828736.212773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828736.212874:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828736.404760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828736.404803:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828736.404815:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1787828736.404817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828736.404821:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828736.404835:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1742 00000100:00080000:3.0:1787828736.404850:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828736.404860:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828736.404932:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672517965952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828736.404973:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828736.404978:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828736.404980:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828736.404986:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1742 00000100:00080000:3.0:1787828736.404991:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828736.404996:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828736.405010:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828736.405014:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517965952:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828736.405033:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672517966080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828736.405058:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828736.405063:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828736.405065:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828736.405071:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1742 00000800:00000100:0.0:1787828736.405078:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828736.405078:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828736.405090:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828736.405098:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828736.405117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672517966208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828736.405128:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff8b2e88039e00 x1874672517965952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828736.405136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828736.405148:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828736.405151:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828736.405152:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517965952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828736.405167:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1742 00000100:00080000:0.0:1787828736.405170:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828736.405176:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828736.405177:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828736.405187:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517965952:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828736.405191:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828736.405202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828736.405213:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517966080:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828736.405222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672517966336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828736.405233:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828736.405247:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828736.405253:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff8b2e88039680 x1874672517966080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828736.405258:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828736.405267:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828736.405275:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517966080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828736.405277:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1742 00000100:00080000:3.0:1787828736.405292:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828736.405309:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.405317:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787828736.405317:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828736.405328:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517966080:192.168.203.101@tcp:38:kworker.0 00000008:00100000:3.0:1787828736.405332:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828736.405337:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517966208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828736.405386:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828736.405389:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672517966464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828736.405401:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff8b2e8803b480 x1874672517966208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828736.405420:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517966208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828736.405424:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828736.405451:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.405459:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787828736.405459:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828736.405469:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517966208:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787828736.405469:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828736.405503:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1742 00000100:00080000:3.0:1787828736.405510:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828736.405515:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828736.405524:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517966336:192.168.203.101@tcp:8:kworker.0 00000008:00100000:3.0:1787828736.405529:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828736.405556:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828736.405557:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672517966592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828736.405567:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff8b2e880392c0 x1874672517966336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828736.405577:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828736.405582:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517966336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828736.405586:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828736.405598:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828736.405598:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828736.405607:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1742 00000100:00080000:0.0:1787828736.405619:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787828736.405620:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828736.405627:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517966336:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787828736.405632:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828736.405638:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517966464:192.168.203.101@tcp:38:kworker.0 00000008:00100000:3.0:1787828736.405642:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828736.405688:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828736.405703:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff8b2e88039e00 x1874672517966464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828736.405707:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517966720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828736.405728:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517966464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828736.405795:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.405801:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828736.405805:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517966464:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828736.405819:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517966592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828736.405836:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828736.405844:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff8b2e8803a1c0 x1874672517966592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828736.405852:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517966592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828736.405857:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.405859:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828736.405861:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517966592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828736.405865:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517966720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828736.405873:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828736.405879:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828736/real 1787828736] req@ffff8b2e880383c0 x1874672517966720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828736.405886:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517966720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828761 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828736.405891:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828736.405893:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828736.405895:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517966720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828736.724718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828737.236778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828737.236845:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828737.428785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828737.428795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828737.748707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828738.260831:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828738.260953:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828738.452723:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828738.452742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828738.772703:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828739.284829:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828739.284904:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828739.476781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828739.476781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828739.796808:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828740.308759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828740.308853:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828740.500749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828740.500805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828740.820766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828741.332762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828741.332891:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828741.524760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828741.524788:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787828741.524793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828741.524805:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828741.524814:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828741.524827:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1747 00000100:00080000:3.0:1787828741.524843:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828741.524853:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828741.524923:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517966848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828741.524968:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828741.524975:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828741.524979:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828741.524987:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1747 00000100:00080000:3.0:1787828741.524995:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828741.525003:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828741.525035:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828741.525062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672517966976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828741.525063:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517966848:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828741.525094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828741.525101:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828741.525106:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828741.525114:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1747 00000100:00080000:3.0:1787828741.525122:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828741.525127:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828741.525134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828741.525151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672517967104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828741.525161:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828741.525185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828741.525189:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828741.525191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828741.525197:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1747 00000100:00080000:3.0:1787828741.525201:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828741.525204:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828741.525219:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828741.525249:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672517967232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828741.525263:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff8b2e8803a580 x1874672517966848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828741.525263:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828741.525273:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828741.525277:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828741.525288:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1747 00000100:00080000:3.0:1787828741.525296:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828741.525305:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828741.525313:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828741.525325:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517966848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828741.525336:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672517967360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828741.525371:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828741.525378:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828741.525383:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787828741.525386:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828741.525391:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828741.525395:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517966848:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828741.525401:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1747 00000100:00080000:3.0:1787828741.525408:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828741.525412:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828741.525417:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828741.525431:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517967488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828741.525437:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517966976:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787828741.525445:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828741.525450:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828741.525451:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828741.525456:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1747 00000100:00080000:3.0:1787828741.525459:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828741.525462:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828741.525466:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828741.525474:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517967616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828741.525485:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828741.525509:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff8b2e8803b840 x1874672517966976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828741.525528:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517966976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.525553:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.525560:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828741.525564:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517966976:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828741.525574:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517967104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828741.525604:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828741.525620:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff8b2e88038f00 x1874672517967104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828741.525636:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517967104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.525676:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.525681:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828741.525687:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517967104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828741.525695:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517967232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828741.525742:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828741.525761:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff8b2e8803b0c0 x1874672517967232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828741.525797:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517967232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.525812:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.525819:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828741.525826:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517967232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828741.525839:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517967360:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828741.525905:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828741.525930:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff8b2e88039a40 x1874672517967360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828741.525955:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517967360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.525972:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.525978:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828741.525984:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517967360:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828741.526036:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517967488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828741.526074:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828741.526094:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff8b2e8803a580 x1874672517967488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828741.526115:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517967488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.526129:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.526134:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828741.526139:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517967488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828741.526165:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517967616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828741.526193:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828741.526208:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828741/real 1787828741] req@ffff8b2e880383c0 x1874672517967616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828741.526243:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517967616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828766 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828741.526268:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828741.526274:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828741.526280:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517967616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828741.844714:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828742.356737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828742.356811:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828742.548787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828742.548793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828742.868785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828743.380825:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828743.380932:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828743.572803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828743.572807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828743.892777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828744.404791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828744.404889:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828744.596774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828744.596907:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828744.916806:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828745.428797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828745.428850:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828745.620794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828745.620810:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828745.940825:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828746.452893:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828746.453043:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828746.644863:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828746.644893:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828746.644904:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828746.644910:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787828746.644926:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828746.644952:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1753 00000100:00080000:3.0:1787828746.644966:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828746.644975:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828746.645040:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672517967744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828746.645070:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828746.645074:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828746.645076:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828746.645080:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1753 00000100:00080000:3.0:1787828746.645084:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828746.645087:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828746.645099:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828746.645113:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672517967872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828746.645124:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828746.645128:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828746.645130:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828746.645134:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1753 00000100:00080000:3.0:1787828746.645137:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828746.645143:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828746.645154:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828746.645156:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517967744:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828746.645180:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672517968000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828746.645196:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828746.645199:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828746.645202:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828746.645206:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1753 00000100:00080000:3.0:1787828746.645212:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828746.645215:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828746.645220:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828746.645235:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672517968128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828746.645256:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828746.645261:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828746.645263:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828746.645267:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1753 00000100:00080000:3.0:1787828746.645270:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828746.645274:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828746.645280:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828746.645297:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672517968256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828746.645301:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828746.645350:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828746.645357:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828746.645360:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828746.645370:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1753 00000100:00080000:3.0:1787828746.645378:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828746.645386:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828746.645394:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828746.645412:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517968384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828746.645424:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828746.645428:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff8b2e8803a1c0 x1874672517967744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828746.645430:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828746.645436:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828746.645456:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1753 00000100:00080000:3.0:1787828746.645475:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828746.645486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517967744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828746.645489:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828746.645515:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828746.645520:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.645529:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828746.645536:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517967744:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828746.645548:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517968512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828746.645551:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517967872:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828746.645632:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828746.645646:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff8b2e88039e00 x1874672517967872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828746.645659:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517967872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.645694:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.645697:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828746.645701:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517967872:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828746.645750:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517968000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828746.645793:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828746.645808:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff8b2e880392c0 x1874672517968000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828746.645824:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517968000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.645877:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.645881:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828746.645884:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517968000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828746.645890:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517968128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828746.645910:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828746.645942:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff8b2e8803b480 x1874672517968128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828746.645957:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517968128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.645968:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.645972:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828746.645976:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517968128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828746.645983:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517968256:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828746.646003:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828746.646014:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff8b2e88039680 x1874672517968256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828746.646029:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517968256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.646040:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.646043:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828746.646047:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517968256:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828746.646054:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517968384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828746.646073:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828746.646084:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff8b2e880383c0 x1874672517968384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828746.646098:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517968384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.646108:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.646112:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828746.646115:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517968384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828746.646122:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517968512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828746.646140:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828746.646150:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828746/real 1787828746] req@ffff8b2e8803a580 x1874672517968512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828746.646164:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517968512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828771 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828746.646183:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828746.646187:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828746.646191:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517968512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828746.964900:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828747.476809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828747.476981:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828747.668785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828747.668873:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828747.988795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828748.500722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828748.500784:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828748.692731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828748.692770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828749.012755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828749.524857:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828749.524958:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828749.716787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828749.716882:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828750.036830:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828750.548781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828750.548906:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828750.740786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828750.740863:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828751.060726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828751.572772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828751.572898:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828751.764805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828751.764805:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828751.764808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828751.764829:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828751.764836:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828751.764847:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1758 00000100:00080000:3.0:1787828751.764859:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828751.764867:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828751.764922:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672517968640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828751.764968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828751.764972:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828751.764974:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828751.764978:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1758 00000100:00080000:3.0:1787828751.764982:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828751.764986:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828751.764999:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828751.765013:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672517968768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828751.765034:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828751.765036:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517968640:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828751.765042:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828751.765063:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828751.765068:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1758 00000100:00080000:3.0:1787828751.765072:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828751.765076:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828751.765082:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828751.765097:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672517968896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828751.765115:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828751.765119:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828751.765121:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828751.765124:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1758 00000100:00080000:3.0:1787828751.765128:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828751.765132:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828751.765136:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828751.765141:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828751.765150:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672517969024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828751.765158:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828751.765161:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828751.765162:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828751.765165:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1758 00000100:00080000:3.0:1787828751.765168:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828751.765170:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828751.765173:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828751.765183:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517969152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828751.765188:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff8b2e88039a40 x1874672517968640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828751.765194:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828751.765202:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828751.765207:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828751.765212:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1758 00000100:00080000:3.0:1787828751.765215:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828751.765217:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517968640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828751.765220:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828751.765227:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828751.765233:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828751.765238:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517969280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828751.765241:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828751.765249:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517968640:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828751.765256:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828751.765259:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828751.765261:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828751.765265:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1758 00000100:00080000:3.0:1787828751.765267:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828751.765270:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828751.765272:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828751.765277:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517968768:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828751.765282:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672517969408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828751.765303:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828751.765315:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff8b2e8803b0c0 x1874672517968768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828751.765324:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517968768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.765330:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.765339:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828751.765342:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517968768:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828751.765348:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517968896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828751.765362:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828751.765369:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff8b2e88038f00 x1874672517968896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828751.765377:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517968896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.765383:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.765385:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828751.765387:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517968896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828751.765391:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517969024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828751.765401:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828751.765406:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff8b2e8803b840 x1874672517969024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828751.765414:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517969024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.765419:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.765421:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828751.765422:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517969024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828751.765427:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517969152:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828751.765437:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828751.765449:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff8b2e8803a580 x1874672517969152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828751.765457:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517969152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.765464:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.765466:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828751.765468:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517969152:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828751.765471:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517969280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828751.765482:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828751.765488:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff8b2e880383c0 x1874672517969280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828751.765496:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517969280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.765501:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.765502:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828751.765504:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517969280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828751.765520:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517969408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828751.765541:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828751.765554:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828751/real 1787828751] req@ffff8b2e88039680 x1874672517969408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828751.765567:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517969408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828776 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828751.765585:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828751.765588:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828751.765590:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517969408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828752.084819:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828752.596758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828752.596859:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828752.788763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828752.788777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828753.108752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828753.620798:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828753.620913:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828753.812759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828753.812761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828754.132837:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828754.644768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828754.644855:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828754.836834:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828754.836909:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828755.156757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828755.668898:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828755.668967:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828755.860742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828755.860857:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828756.180815:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828756.692823:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828756.692836:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828756.884770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828756.884770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828756.884803:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828756.884810:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828756.884814:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828756.884825:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1763 00000100:00080000:0.0:1787828756.884837:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828756.884845:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828756.884888:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517969536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828756.884914:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828756.884917:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828756.884919:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828756.884922:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1763 00000100:00080000:0.0:1787828756.884925:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828756.884927:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828756.884947:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828756.884958:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517969664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828756.884966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828756.884969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828756.884971:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828756.884976:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1763 00000100:00100000:3.0:1787828756.884977:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517969536:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828756.884980:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828756.884987:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828756.884991:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828756.884999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517969792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828756.885006:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828756.885008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828756.885010:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828756.885012:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1763 00000100:00080000:0.0:1787828756.885014:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828756.885017:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828756.885019:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828756.885026:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517969920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828756.885032:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828756.885034:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828756.885036:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828756.885039:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1763 00000100:00080000:0.0:1787828756.885049:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828756.885052:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828756.885055:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828756.885061:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828756.885067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517970048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828756.885075:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828756.885077:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828756.885079:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828756.885082:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1763 00000100:00080000:0.0:1787828756.885086:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828756.885088:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828756.885092:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828756.885099:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517970176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828756.885106:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828756.885108:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828756.885113:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828756.885113:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff8b2e88042580 x1874672517969536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828756.885118:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1763 00000100:00080000:0.0:1787828756.885123:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828756.885128:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828756.885145:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828756.885155:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672517969536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828756.885156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517970304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828756.885191:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828756.885196:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828756.885211:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517969536:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828756.885232:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517969664:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828756.885248:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828756.885259:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff8b2e88041e00 x1874672517969664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828756.885273:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517969664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828756.885282:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828756.885286:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828756.885290:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517969664:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828756.885297:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517969792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828756.885318:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828756.885328:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff8b2e88040b40 x1874672517969792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828756.885336:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517969792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828756.885347:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828756.885349:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828756.885351:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517969792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828756.885355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517969920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828756.885367:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828756.885373:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff8b2e880421c0 x1874672517969920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828756.885381:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517969920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828756.885387:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828756.885389:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828756.885391:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517969920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828756.885394:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517970048:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828756.885404:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828756.885410:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff8b2e88042d00 x1874672517970048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828756.885417:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517970048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828756.885423:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828756.885425:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828756.885428:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517970048:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828756.885432:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517970176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828756.885442:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828756.885447:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff8b2e88040780 x1874672517970176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828756.885454:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517970176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828756.885460:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828756.885461:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828756.885463:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517970176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828756.885466:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517970304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828756.885476:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828756.885482:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828756/real 1787828756] req@ffff8b2e88043840 x1874672517970304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828756.885490:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517970304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828781 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828756.885495:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828756.885496:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828756.885498:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517970304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828757.204730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828757.716733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828757.716898:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828757.908746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828757.908746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828758.228885:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828758.740772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828758.740783:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828758.932761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828758.932762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828759.252727:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828759.764729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828759.764800:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828759.956812:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828759.956812:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828760.276831:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828760.788837:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828760.788939:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828760.980796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828760.980796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828761.300818:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828761.812786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828761.812880:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828762.004812:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828762.004812:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828762.004850:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828762.004864:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828762.004889:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828762.004908:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1768 00000100:00080000:0.0:1787828762.004928:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828762.004941:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828762.005015:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517970432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828762.005058:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828762.005064:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828762.005067:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828762.005074:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1768 00000100:00080000:0.0:1787828762.005080:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828762.005087:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828762.005102:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828762.005118:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517970432:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828762.005127:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517970560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828762.005146:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828762.005152:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828762.005155:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828762.005161:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1768 00000100:00080000:0.0:1787828762.005168:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828762.005174:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828762.005182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828762.005201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517970688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828762.005210:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828762.005236:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828762.005241:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828762.005245:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828762.005252:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1768 00000100:00080000:0.0:1787828762.005259:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828762.005265:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828762.005272:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828762/real 1787828762] req@ffff8b2e880412c0 x1874672517970432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828762.005275:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828762.005300:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517970816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828762.005303:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672517970432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828762.005342:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828762.005343:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828762.005354:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787828762.005355:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828762.005362:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828762.005363:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517970432:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828762.005379:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1768 00000100:00080000:0.0:1787828762.005387:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828762.005393:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828762.005401:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517970560:192.168.203.101@tcp:38:kworker.0 00000008:00100000:0.0:1787828762.005402:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828762.005429:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517970944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828762.005446:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828762.005451:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828762.005456:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828762.005461:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828762.005469:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1768 00000100:00100000:3.0:1787828762.005475:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828762/real 1787828762] req@ffff8b2e88041a40 x1874672517970560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828762.005479:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828762.005492:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828762.005496:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517970560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828762.005504:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828762.005508:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828762.005514:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828762.005518:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517970560:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828762.005529:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517971072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828762.005530:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517970688:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787828762.005553:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828762.005559:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787828762.005564:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828762.005566:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828762.005576:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1768 00000100:00100000:3.0:1787828762.005581:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828762/real 1787828762] req@ffff8b2e88040f00 x1874672517970688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828762.005585:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828762.005597:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828762.005605:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517970688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828762.005608:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828762.005623:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828762.005627:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828762.005637:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517971200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828762.005637:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517970688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828762.005674:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517970816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828762.005707:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828762.005751:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828762/real 1787828762] req@ffff8b2e88042940 x1874672517970816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828762.005772:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517970816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828762.005784:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828762.005787:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828762.005790:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517970816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828762.005807:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517970944:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828762.005829:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828762.005841:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828762/real 1787828762] req@ffff8b2e88043480 x1874672517970944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828762.005852:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517970944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828762.005861:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828762.005863:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828762.005866:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517970944:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828762.005872:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517971072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828762.005886:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828762.005894:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828762/real 1787828762] req@ffff8b2e880430c0 x1874672517971072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828762.005904:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517971072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828762.005911:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828762.005924:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828762.005928:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517971072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828762.005933:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517971200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828762.005961:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828762.005971:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828762/real 1787828762] req@ffff8b2e88040000 x1874672517971200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828762.005982:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517971200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828787 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828762.005990:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828762.005993:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828762.005996:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517971200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828762.324802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828762.836863:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828762.836925:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828763.028731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828763.028748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828763.348727:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828763.860784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828763.860807:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828764.052804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828764.052813:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828764.372811:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828764.884818:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828764.884888:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828765.076809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828765.076811:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828765.396751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828765.908766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828765.908836:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828766.100733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828766.100733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828766.420764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828766.932798:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828766.932833:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828767.124743:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828767.124743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828767.124743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828767.124754:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828767.124758:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828767.124768:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1773 00000100:00080000:0.0:1787828767.124778:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828767.124784:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828767.124818:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517971328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828767.124844:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828767.124847:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828767.124848:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828767.124851:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1773 00000100:00080000:0.0:1787828767.124854:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828767.124858:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828767.124867:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828767.124875:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517971328:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828767.124881:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517971456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828767.124896:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828767.124899:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828767.124900:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828767.124904:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1773 00000100:00080000:0.0:1787828767.124908:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828767.124912:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828767.124917:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828767.124931:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517971584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828767.124941:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828767.124944:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828767.124950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828767.124952:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828767.124958:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1773 00000100:00080000:0.0:1787828767.124963:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828767.124967:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828767.124971:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828767.124983:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828767/real 1787828767] req@ffff8b2e880403c0 x1874672517971328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828767.124992:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517971712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828767.125001:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672517971328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828767.125009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828767.125014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828767.125016:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828767.125017:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828767.125026:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787828767.125027:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1773 00000100:00080000:0.0:1787828767.125033:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828767.125036:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517971328:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828767.125040:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828767.125045:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828767.125054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517971840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828767.125061:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517971456:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828767.125063:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828767.125070:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828767.125071:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828767.125075:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1773 00000100:00080000:0.0:1787828767.125079:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828767.125082:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828767.125085:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828767.125091:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828767.125096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517971968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828767.125108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828767.125113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787828767.125113:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828767/real 1787828767] req@ffff8b2e88040000 x1874672517971456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828767.125114:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828767.125120:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1773 00000100:00080000:0.0:1787828767.125124:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828767.125130:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828767.125134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828767.125145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517972096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828767.125149:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517971456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828767.125164:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828767.125169:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828767.125173:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517971456:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828767.125181:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517971584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828767.125201:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828767.125210:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828767/real 1787828767] req@ffff8b2e880430c0 x1874672517971584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828767.125219:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517971584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828767.125224:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828767.125226:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828767.125228:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517971584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828767.125232:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517971712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828767.125242:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828767.125248:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828767/real 1787828767] req@ffff8b2e88043480 x1874672517971712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828767.125255:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517971712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828767.125260:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828767.125262:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828767.125264:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517971712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828767.125275:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517971840:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828767.125285:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828767.125291:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828767/real 1787828767] req@ffff8b2e88042940 x1874672517971840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828767.125298:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517971840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828767.125304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828767.125306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828767.125307:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517971840:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828767.125311:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517971968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828767.125321:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828767.125326:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828767/real 1787828767] req@ffff8b2e88040f00 x1874672517971968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828767.125334:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517971968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828767.125338:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828767.125340:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828767.125342:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517971968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828767.125346:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517972096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828767.125355:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828767.125360:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828767/real 1787828767] req@ffff8b2e88041a40 x1874672517972096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828767.125367:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517972096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828792 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828767.125372:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828767.125374:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828767.125376:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517972096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828767.444803:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828767.956761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828767.956816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828768.148724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828768.148751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828768.468781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828768.980786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828768.980886:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828769.172774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828769.172801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828769.492741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828770.004808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828770.004855:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828770.196796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828770.196796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828770.516793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828771.028778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828771.028872:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828771.220733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828771.220733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828771.540768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828772.052790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828772.053002:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828772.244807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828772.244833:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828772.244835:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828772.244843:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828772.244860:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828772.244872:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1778 00000100:00080000:1.0:1787828772.244886:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828772.244905:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828772.244963:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517972224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828772.244989:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828772.244992:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828772.244993:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828772.244996:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1778 00000100:00080000:1.0:1787828772.245000:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828772.245003:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828772.245012:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828772.245023:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517972352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828772.245031:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828772.245034:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828772.245035:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828772.245038:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1778 00000100:00080000:1.0:1787828772.245041:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828772.245045:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828772.245050:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828772.245054:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517972224:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828772.245067:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517972480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828772.245084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828772.245088:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828772.245091:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828772.245095:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1778 00000100:00080000:1.0:1787828772.245100:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828772.245104:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828772.245109:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828772.245120:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517972608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828772.245132:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828772.245135:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828772.245137:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828772.245140:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1778 00000100:00080000:1.0:1787828772.245143:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828772.245145:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828772.245149:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828772.245154:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828772.245159:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517972736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828772.245173:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828772.245176:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828772.245177:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828772.245181:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1778 00000100:00080000:1.0:1787828772.245183:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828772.245185:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828772.245188:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828772.245194:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517972864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828772.245201:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828772.245204:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828772.245205:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828772.245207:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1778 00000100:00080000:1.0:1787828772.245210:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828772.245213:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828772.245216:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828772.245223:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517972992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828772.245256:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff8b2e88077840 x1874672517972224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828772.245307:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517972224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828772.245319:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828772.245324:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828772.245329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517972224:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828772.245338:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517972352:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828772.245375:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828772.245390:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff8b2e88074780 x1874672517972352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828772.245404:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517972352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828772.245413:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828772.245417:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828772.245421:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517972352:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828772.245462:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517972480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828772.245486:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828772.245499:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff8b2e88076940 x1874672517972480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828772.245512:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517972480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828772.245528:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828772.245531:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828772.245535:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517972480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828772.245541:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517972608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828772.245561:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828772.245575:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff8b2e88074f00 x1874672517972608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828772.245587:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517972608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828772.245596:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828772.245600:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828772.245603:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517972608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828772.245610:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517972736:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828772.245631:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828772.245644:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff8b2e88075680 x1874672517972736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828772.245698:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517972736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828772.245718:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828772.245721:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828772.245726:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517972736:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828772.245733:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517972864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828772.245756:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828772.245770:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff8b2e88075a40 x1874672517972864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828772.245784:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517972864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828772.245793:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828772.245797:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828772.245800:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517972864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828772.245807:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517972992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828772.245827:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828772.245838:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828772/real 1787828772] req@ffff8b2e88077480 x1874672517972992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828772.245851:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517972992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828797 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828772.245859:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828772.245863:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828772.245867:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517972992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828772.564772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828773.076753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828773.076784:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828773.268728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828773.268772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828773.588743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828774.100774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828774.100843:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828774.292761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828774.292761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828774.612739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828775.124758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828775.124840:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828775.316712:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828775.316735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828775.636727:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828776.148799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828776.148845:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828776.340769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828776.340770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828776.660769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828777.172773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828777.172871:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828777.364729:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828777.364730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828777.364730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828777.364739:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828777.364744:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828777.364755:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1783 00000100:00080000:1.0:1787828777.364766:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828777.364774:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828777.364814:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517973120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828777.364837:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828777.364840:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828777.364841:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828777.364844:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1783 00000100:00080000:1.0:1787828777.364848:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828777.364851:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828777.364862:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828777.364871:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517973120:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828777.364875:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517973248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828777.364888:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828777.364894:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828777.364895:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828777.364898:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1783 00000100:00080000:1.0:1787828777.364901:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828777.364904:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828777.364909:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828777.364922:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517973376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828777.364933:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828777.364937:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828777.364940:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828777.364945:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1783 00000100:00080000:1.0:1787828777.364950:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828777.364955:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787828777.364959:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787828777.364965:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828777.364979:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517973504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828777.364986:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828777.364989:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828777.364990:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828777.364994:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1783 00000100:00080000:1.0:1787828777.364997:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828777.365001:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828777.365007:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff8b2e88077c00 x1874672517973120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828777.365013:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828777.365026:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517973632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828777.365027:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517973120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828777.365054:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828777.365056:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828777.365060:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828777.365065:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787828777.365066:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517973120:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828777.365071:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828777.365079:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1783 00000100:00080000:1.0:1787828777.365083:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828777.365089:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828777.365091:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517973248:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787828777.365095:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828777.365111:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517973760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828777.365120:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828777.365121:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828777.365126:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828777.365128:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828777.365133:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1783 00000100:00080000:1.0:1787828777.365137:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828777.365143:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828777.365144:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff8b2e88077480 x1874672517973248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828777.365147:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828777.365159:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517973888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828777.365196:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517973248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828777.365203:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828777.365206:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828777.365208:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517973248:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828777.365213:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517973376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828777.365231:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828777.365239:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff8b2e88075a40 x1874672517973376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828777.365247:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517973376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828777.365253:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828777.365255:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828777.365257:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517973376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828777.365261:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517973504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828777.365270:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828777.365276:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff8b2e88075680 x1874672517973504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828777.365283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517973504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828777.365294:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828777.365296:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828777.365298:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517973504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828777.365301:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517973632:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828777.365311:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828777.365317:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff8b2e88074f00 x1874672517973632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828777.365324:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517973632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828777.365330:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828777.365332:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828777.365333:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517973632:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828777.365345:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517973760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828777.365355:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828777.365361:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff8b2e88076940 x1874672517973760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828777.365369:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517973760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828777.365374:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828777.365376:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828777.365378:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517973760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828777.365382:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517973888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828777.365392:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828777.365397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828777/real 1787828777] req@ffff8b2e88074780 x1874672517973888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828777.365405:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517973888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828802 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828777.365410:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828777.365412:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828777.365413:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517973888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828777.684731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828778.196744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828778.196873:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828778.388716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828778.388716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828778.708779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828779.220744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828779.220904:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828779.412739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828779.412739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828779.732731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828780.244757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828780.244816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828780.436717:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828780.436717:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828780.756766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828781.268735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828781.268811:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828781.460726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828781.460726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828781.780778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828782.292778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828782.292820:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828782.484786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828782.484786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828782.484814:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828782.484820:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828782.484825:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828782.484836:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1788 00000100:00080000:0.0:1787828782.484860:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828782.484868:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828782.484929:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517974016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828782.484951:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828782.484954:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828782.484956:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828782.484960:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1788 00000100:00080000:0.0:1787828782.484963:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828782.484966:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828782.484976:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828782.484990:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517974016:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828782.484991:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517974144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828782.485006:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828782.485008:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828782.485009:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828782.485013:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1788 00000100:00080000:0.0:1787828782.485015:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828782.485017:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828782.485021:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828782.485031:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517974272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828782.485037:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828782.485040:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828782.485042:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828782.485044:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1788 00000100:00080000:0.0:1787828782.485047:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828782.485049:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828782.485052:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828782.485060:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517974400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828782.485067:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828782.485071:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828782.485072:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828782.485075:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1788 00000800:00000100:3.0:1787828782.485080:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828782.485081:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828782.485087:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828782.485091:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828782.485100:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517974528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828782.485107:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828782.485110:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828782.485111:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828782.485114:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1788 00000100:00080000:0.0:1787828782.485116:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828782.485119:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828782.485123:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828782.485129:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff8b2e880412c0 x1874672517974016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828782.485139:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517974656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828782.485148:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672517974016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828782.485152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828782.485160:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828782.485161:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828782.485163:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828782.485167:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787828782.485171:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1788 00000100:00100000:3.0:1787828782.485173:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517974016:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828782.485175:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828782.485183:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828782.485186:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828782.485194:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517974784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828782.485206:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517974144:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828782.485236:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828782.485251:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff8b2e88043840 x1874672517974144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828782.485261:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517974144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828782.485267:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828782.485270:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828782.485272:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517974144:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828782.485277:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517974272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828782.485292:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828782.485299:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff8b2e88040780 x1874672517974272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828782.485312:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517974272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828782.485317:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828782.485319:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828782.485321:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517974272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828782.485325:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517974400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828782.485335:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828782.485340:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff8b2e88042d00 x1874672517974400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828782.485348:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517974400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828782.485353:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828782.485355:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828782.485357:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517974400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828782.485361:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517974528:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828782.485371:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828782.485377:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff8b2e880421c0 x1874672517974528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828782.485384:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517974528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828782.485394:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828782.485396:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828782.485399:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517974528:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828782.485402:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517974656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828782.485412:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828782.485417:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff8b2e88040b40 x1874672517974656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828782.485424:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517974656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828782.485438:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828782.485441:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828782.485443:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517974656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828782.485456:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517974784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828782.485469:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828782.485476:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828782/real 1787828782] req@ffff8b2e88041e00 x1874672517974784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828782.485484:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517974784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828807 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828782.485490:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828782.485492:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828782.485495:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517974784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828782.804756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828783.316747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828783.316835:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828783.508775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828783.508776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828783.828745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828784.340732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828784.340819:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828784.532737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828784.532738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828784.852715:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828785.364736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828785.364764:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828785.556720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828785.556720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828785.876730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828786.388747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828786.388839:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828786.580783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828786.580783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828786.900762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828787.412778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828787.412783:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828787.604748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828787.604753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828787.604771:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828787.604780:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828787.604785:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828787.604794:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1794 00000100:00080000:1.0:1787828787.604804:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828787.604810:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828787.604846:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517974912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828787.604869:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828787.604872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828787.604874:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828787.604877:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1794 00000100:00080000:1.0:1787828787.604880:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828787.604884:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828787.604893:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828787.604903:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517975040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828787.604912:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517974912:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828787.604920:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828787.604924:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828787.604925:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828787.604929:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1794 00000100:00080000:1.0:1787828787.604931:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828787.604935:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828787.604938:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828787.604952:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517975168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828787.604970:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828787.604974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828787.604976:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828787.604981:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1794 00000100:00080000:1.0:1787828787.604986:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828787.604991:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787828787.604997:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787828787.604998:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828787.605011:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517975296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828787.605019:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828787.605022:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828787.605023:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828787.605027:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1794 00000100:00080000:1.0:1787828787.605030:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828787.605033:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828787.605037:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828787.605057:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff8b2e88077840 x1874672517974912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828787.605059:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517975424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828787.605130:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828787.605134:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828787.605136:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828787.605142:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1794 00000100:00100000:3.0:1787828787.605144:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517974912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828787.605149:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828787.605163:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828787.605179:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828787.605181:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828787.605187:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787828787.605192:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517975552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828787.605193:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517974912:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828787.605216:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828787.605218:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828787.605220:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828787.605223:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1794 00000100:00080000:1.0:1787828787.605226:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828787.605228:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828787.605233:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828787.605237:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517975040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828787.605245:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517975680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828787.605264:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828787.605283:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff8b2e880770c0 x1874672517975040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828787.605292:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517975040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828787.605299:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828787.605301:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828787.605304:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517975040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828787.605308:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517975168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828787.605321:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828787.605327:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff8b2e88076580 x1874672517975168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828787.605335:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517975168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828787.605340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828787.605342:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828787.605344:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517975168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828787.605348:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517975296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828787.605358:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828787.605364:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff8b2e88076d00 x1874672517975296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828787.605376:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517975296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828787.605382:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828787.605384:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828787.605385:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517975296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828787.605389:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517975424:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828787.605400:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828787.605406:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff8b2e880761c0 x1874672517975424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828787.605413:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517975424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828787.605419:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828787.605420:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828787.605422:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517975424:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828787.605426:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517975552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828787.605436:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828787.605442:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff8b2e88075e00 x1874672517975552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828787.605449:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517975552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828787.605459:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828787.605461:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828787.605463:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517975552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828787.605479:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517975680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828787.605500:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828787.605512:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828787/real 1787828787] req@ffff8b2e88074b40 x1874672517975680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828787.605525:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517975680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828812 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828787.605534:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828787.605538:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828787.605555:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517975680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828787.924738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828788.436755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828788.436807:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828788.628715:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828788.628718:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828788.948753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828789.460731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828789.460839:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828789.652783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828789.652785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828789.972780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828790.484792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828790.484867:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828790.676741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828790.676741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828790.996792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828791.508727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828791.508837:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828791.700735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828791.700788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828792.020732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828792.532751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828792.532829:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828792.724766:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828792.724766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828792.724775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828792.724777:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828792.724782:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828792.724793:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1799 00000100:00080000:1.0:1787828792.724805:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828792.724812:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828792.724853:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517975808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828792.724875:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828792.724879:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828792.724881:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828792.724884:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1799 00000100:00080000:1.0:1787828792.724887:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828792.724890:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828792.724898:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828792.724909:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517975808:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828792.724913:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517975936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828792.724923:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828792.724926:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828792.724927:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828792.724930:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1799 00000100:00080000:1.0:1787828792.724933:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828792.724935:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828792.724938:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828792.724946:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517976064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828792.724952:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828792.724955:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828792.724956:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828792.724958:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1799 00000100:00080000:1.0:1787828792.724961:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828792.724963:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828792.724968:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828792.724976:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517976192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828792.724982:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828792.724985:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828792.724986:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828792.724989:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1799 00000100:00080000:1.0:1787828792.724991:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828792.724994:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828792.724998:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828792.725006:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517976320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828792.725014:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517975936:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828792.725020:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828792.725025:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828792.725026:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828792.725030:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1799 00000100:00080000:1.0:1787828792.725033:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828792.725035:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828792.725039:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828792.725049:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517976448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828792.725053:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517976064:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787828792.725064:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828792.725068:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828792.725070:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828792.725073:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1799 00000100:00080000:1.0:1787828792.725075:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828792.725078:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828792.725081:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828792.725088:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517976192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828792.725096:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517976576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828792.725109:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517976320:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828792.725126:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517976448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828792.725143:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517976576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828793.044790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787828793.556831:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828793.556867:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787828793.748754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828793.748768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828794.068757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828794.580729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828794.580786:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828794.772713:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828794.772715:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828795.092750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828795.604826:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828795.604920:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828795.796805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828795.796824:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:2.0:1787828795.797002:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787828795.797051:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828795.797069:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828795.797112:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828795.797123:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828795.797133:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828795.797145:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828795.797157:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787828795.797157:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828795] req@ffff8b2e88074b40 x1874672517975936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828795.797171:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787828795.797190:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517975936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828795.797205:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828795.797211:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828795.797216:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517975936:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828795.797224:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828795] req@ffff8b2e88075e00 x1874672517976064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828795.797234:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517976064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828795.797241:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828795.797245:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828795.797248:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517976064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828795.797255:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828795] req@ffff8b2e880761c0 x1874672517976192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828795.797270:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517976192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828795.797278:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828795.797281:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828795.797285:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517976192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828795.797291:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828795] req@ffff8b2e88074000 x1874672517975808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828795.797304:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517975808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828795.797312:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828795.797315:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828795.797318:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517975808:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828795.797351:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828795] req@ffff8b2e880770c0 x1874672517976576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828795.797363:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517976576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828795.797370:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828795.797374:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828795.797377:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517976576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828795.797383:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828795] req@ffff8b2e88076d00 x1874672517976320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828795.797394:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517976320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828795.797401:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828795.797405:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828795.797409:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517976320:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828795.797415:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828792/real 1787828795] req@ffff8b2e88076580 x1874672517976448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828795.797426:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517976448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828817 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828795.797434:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828795.797437:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828795.797441:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517976448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828796.116822:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828796.628771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828796.628838:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828796.820818:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828796.820824:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828797.140815:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828797.652754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828797.652797:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828797.844785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828797.844786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828797.844840:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.844846:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828797.844849:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.844858:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1804 00000100:00080000:1.0:1787828797.844869:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.844878:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828797.844909:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517976704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828797.844944:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.844948:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828797.844950:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.844955:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1804 00000100:00080000:1.0:1787828797.844959:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.844964:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828797.844974:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828797.844989:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517976704:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828797.844995:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517976832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828797.845009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.845013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828797.845015:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.845019:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1804 00000100:00080000:1.0:1787828797.845024:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.845029:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828797.845033:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828797.845045:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517976960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828797.845051:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828797.845059:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.845062:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828797.845063:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.845067:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1804 00000100:00080000:1.0:1787828797.845071:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.845074:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828797.845077:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828797.845086:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517977088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828797.845092:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.845095:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828797.845096:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.845100:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1804 00000100:00080000:1.0:1787828797.845103:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.845108:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828797.845109:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff8b2e88077840 x1874672517976704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828797.845112:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828797.845125:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517977216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828797.845132:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517976704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828797.845161:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828797.845162:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828797.845166:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828797.845170:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787828797.845172:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517976704:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828797.845175:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828797.845183:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1804 00000100:00080000:1.0:1787828797.845187:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.845191:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828797.845196:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828797.845198:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517976832:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828797.845213:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517977344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828797.845224:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828797.845228:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828797.845232:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787828797.845233:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828797.845240:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1804 00000100:00080000:1.0:1787828797.845247:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828797.845252:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828797.845255:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff8b2e88074780 x1874672517976832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828797.845259:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828797.845282:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517977472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828797.845287:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517976832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828797.845311:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828797.845314:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828797.845318:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517976832:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828797.845327:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517976960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828797.845353:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828797.845367:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff8b2e88076940 x1874672517976960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828797.845380:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517976960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828797.845389:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828797.845392:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828797.845396:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517976960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828797.845402:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517977088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828797.845425:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828797.845437:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff8b2e88074f00 x1874672517977088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828797.845450:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517977088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828797.845465:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828797.845483:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828797.845487:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517977088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828797.845492:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517977216:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828797.845509:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828797.845518:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff8b2e88075680 x1874672517977216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828797.845526:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517977216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828797.845532:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828797.845637:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828797.845642:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517977216:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828797.845696:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517977344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828797.845723:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828797.845737:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff8b2e88075a40 x1874672517977344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828797.845749:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517977344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828797.845758:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828797.845761:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828797.845765:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517977344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828797.845771:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517977472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828797.845791:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828797.845802:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828797/real 1787828797] req@ffff8b2e88077480 x1874672517977472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828797.845815:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517977472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828822 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828797.845827:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828797.845829:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828797.845831:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517977472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828798.164750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828798.676733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828798.676855:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828798.868740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828798.868749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828799.188788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828799.700760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828799.700852:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828799.892747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828799.892747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828800.212692:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828800.724783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828800.724842:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828800.916805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828800.916813:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828801.236769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828801.748810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828801.748870:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828801.940776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828801.940776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828802.260736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828802.772867:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828802.772993:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828802.964779:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828802.964783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828802.964797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828802.964800:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828802.964807:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828802.964820:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1809 00000100:00080000:1.0:1787828802.964834:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828802.964842:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828802.964889:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672517977600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828802.964924:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828802.964928:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828802.964931:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828802.964936:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1809 00000100:00080000:1.0:1787828802.964940:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828802.964944:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828802.964956:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828802.964977:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517977600:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828802.964978:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517977728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828802.965026:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828802.965031:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828802.965033:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828802.965039:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1809 00000100:00080000:1.0:1787828802.965044:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828802.965049:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828802.965058:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828802.965073:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517977856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828802.965087:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787828802.965092:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828802.965098:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828802.965101:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828802.965110:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1809 00000100:00080000:1.0:1787828802.965114:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828802.965120:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828802.965128:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828802.965146:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517977984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828802.965156:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828802.965163:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff8b2e88077c00 x1874672517977600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828802.965165:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828802.965169:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828802.965185:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1809 00000100:00080000:1.0:1787828802.965197:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828802.965203:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828802.965208:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672517977600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828802.965213:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828802.965235:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517978112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828802.965238:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828802.965259:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828802.965261:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828802.965270:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828802.965275:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828802.965282:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517977600:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828802.965284:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1809 00000100:00080000:1.0:1787828802.965301:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828802.965312:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828802.965320:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828802.965329:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517977728:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828802.965344:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517978240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828802.965366:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828802.965369:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828802.965377:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828802.965398:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828802.965405:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff8b2e88077480 x1874672517977728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828802.965409:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1809 00000100:00080000:1.0:1787828802.965420:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828802.965430:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828802.965452:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517977728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828802.965457:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828802.965489:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828802.965498:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517978368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828802.965499:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828802.965520:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517977728:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828802.965545:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517977856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828802.965577:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828802.965589:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff8b2e88075a40 x1874672517977856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828802.965603:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517977856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828802.965610:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828802.965613:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828802.965615:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517977856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828802.965621:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517977984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828802.965633:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828802.965639:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff8b2e88075680 x1874672517977984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828802.965671:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517977984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828802.965677:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828802.965680:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828802.965682:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517977984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828802.965686:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517978112:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828802.965701:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828802.965708:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff8b2e88074f00 x1874672517978112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828802.965715:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517978112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828802.965727:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828802.965729:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828802.965731:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517978112:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828802.965745:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517978240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828802.965756:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828802.965763:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff8b2e88076940 x1874672517978240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828802.965771:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517978240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828802.965788:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828802.965790:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828802.965792:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517978240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828802.965797:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517978368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828802.965807:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828802.965813:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828802/real 1787828802] req@ffff8b2e88074780 x1874672517978368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828802.965820:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517978368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828827 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828802.965826:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828802.965827:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828802.965830:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517978368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828803.284748:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828803.796755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828803.796811:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828803.988755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828803.988755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828804.308704:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828804.820763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828804.820860:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828805.012765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828805.012778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828805.332710:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828805.844835:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828805.844957:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828806.036784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828806.036785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828806.356816:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828806.868730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828806.868788:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828807.060769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828807.060775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828807.380824:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828807.892740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828807.892830:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828808.084725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828808.084725:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828808.084738:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828808.084743:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828808.084747:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828808.084756:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1814 00000100:00080000:0.0:1787828808.084765:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828808.084772:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828808.084806:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517978496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828808.084829:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828808.084832:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828808.084833:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828808.084836:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1814 00000100:00080000:0.0:1787828808.084838:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828808.084840:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828808.084848:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828808.084859:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517978624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828808.084872:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828808.084874:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517978496:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828808.084875:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828808.084878:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828808.084885:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1814 00000100:00080000:0.0:1787828808.084887:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828808.084890:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828808.084893:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828808.084900:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517978752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828808.084906:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828808.084908:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828808.084909:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828808.084911:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1814 00000100:00080000:0.0:1787828808.084914:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828808.084916:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828808.084919:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828808.084926:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517978880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828808.084931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828808.084933:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828808.084934:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828808.084937:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1814 00000100:00080000:0.0:1787828808.084940:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828808.084946:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828808.084953:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828808.084957:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828808.084966:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517979008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828808.084972:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828808.084974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828808.084976:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828808.084978:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1814 00000100:00080000:0.0:1787828808.084981:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828808.084984:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828808.084988:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828808.084992:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828808/real 1787828808] req@ffff8b2e88042580 x1874672517978496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828808.084999:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517979136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828808.085007:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828808.085012:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787828808.085012:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672517978496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828808.085016:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828808.085023:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1814 00000100:00080000:0.0:1787828808.085028:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828808.085030:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828808.085031:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828808.085038:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828808.085039:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828808.085045:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517978496:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828808.085048:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517979264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828808.085068:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517978624:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828808.085089:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828808.085097:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828808/real 1787828808] req@ffff8b2e88041e00 x1874672517978624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828808.085106:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517978624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828808.085111:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828808.085114:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828808.085116:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517978624:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828808.085120:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517978752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828808.085130:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828808.085136:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828808/real 1787828808] req@ffff8b2e88040b40 x1874672517978752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828808.085143:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517978752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828808.085148:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828808.085150:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828808.085152:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517978752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828808.085155:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517978880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828808.085165:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828808.085170:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828808/real 1787828808] req@ffff8b2e880421c0 x1874672517978880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828808.085182:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517978880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828808.085187:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828808.085189:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828808.085190:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517978880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828808.085194:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517979008:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828808.085203:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828808.085210:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828808/real 1787828808] req@ffff8b2e88042d00 x1874672517979008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828808.085217:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517979008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828808.085222:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828808.085224:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828808.085225:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517979008:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828808.085229:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517979136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828808.085238:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828808.085244:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828808/real 1787828808] req@ffff8b2e88040780 x1874672517979136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828808.085251:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517979136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828808.085260:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828808.085262:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828808.085264:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517979136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828808.085267:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517979264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828808.085276:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828808.085281:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828808/real 1787828808] req@ffff8b2e88043840 x1874672517979264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828808.085288:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517979264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828833 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828808.085293:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828808.085294:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828808.085296:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517979264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828808.404736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828808.916730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828808.916798:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828809.108778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828809.108785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828809.428789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828809.940768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828809.940839:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828810.132767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828810.132795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828810.452778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828810.964724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828810.964809:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828811.156809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828811.156809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828811.476743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787828811.988792:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828811.988803:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787828812.180742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828812.180792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828812.500807:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828813.012782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828813.012890:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828813.204812:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828813.204816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828813.204817:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828813.204828:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828813.204835:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828813.204849:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1819 00000100:00080000:0.0:1787828813.204864:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828813.204874:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828813.204930:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517979392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828813.204971:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828813.204977:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828813.204980:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828813.204988:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1819 00000100:00080000:0.0:1787828813.204995:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828813.205002:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828813.205018:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828813.205020:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517979392:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828813.205069:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517979520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828813.205091:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828813.205098:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828813.205101:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828813.205108:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1819 00000100:00080000:0.0:1787828813.205115:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828813.205122:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828813.205133:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787828813.205147:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828813.205157:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517979648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828813.205179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828813.205197:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828813.205200:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828813.205209:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1819 00000100:00080000:0.0:1787828813.205217:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828813.205226:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828813/real 1787828813] req@ffff8b2e880412c0 x1874672517979392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828813.205228:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828813.205243:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828813.205263:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672517979392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828813.205268:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517979776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828813.205311:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828813.205312:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828813.205323:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828813.205326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787828813.205328:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828813.205339:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517979392:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828813.205341:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1819 00000100:00080000:0.0:1787828813.205353:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828813.205361:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828813.205415:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828813.205429:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517979904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828813.205440:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517979520:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828813.205443:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828813.205470:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828813.205473:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828813.205480:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1819 00000100:00080000:0.0:1787828813.205485:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828813.205489:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828813.205497:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828813.205516:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517980032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828813.205519:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828813.205535:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828813.205540:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828813.205543:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828813.205547:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1819 00000100:00080000:0.0:1787828813.205553:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828813.205562:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828813.205564:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828813/real 1787828813] req@ffff8b2e88041a40 x1874672517979520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828813.205568:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828813.205585:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517980160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828813.205620:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517979520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828813.205642:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828813.205650:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828813.205657:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517979520:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828813.205712:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517979648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828813.205774:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828813.205794:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828813/real 1787828813] req@ffff8b2e88040f00 x1874672517979648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828813.205812:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517979648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828813.205824:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828813.205829:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828813.205834:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517979648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828813.205844:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517979776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828813.205876:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828813.205897:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828813/real 1787828813] req@ffff8b2e88042940 x1874672517979776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828813.205912:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517979776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828813.205921:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828813.205925:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828813.205928:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517979776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828813.205947:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517979904:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828813.205969:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828813.205981:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828813/real 1787828813] req@ffff8b2e88043480 x1874672517979904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828813.206001:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517979904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828813.206010:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828813.206034:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828813.206037:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517979904:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828813.206044:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517980032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828813.206064:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828813.206075:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828813/real 1787828813] req@ffff8b2e880430c0 x1874672517980032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828813.206089:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517980032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828813.206098:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828813.206101:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828813.206104:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517980032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828813.206110:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517980160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828813.206126:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828813.206135:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828813/real 1787828813] req@ffff8b2e88040000 x1874672517980160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828813.206148:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517980160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828838 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828813.206164:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828813.206168:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828813.206171:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517980160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828813.524752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828814.036726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828814.036804:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828814.228783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828814.228858:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828814.548768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828815.060802:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828815.060860:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828815.252797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828815.252822:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828815.572729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828816.084736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828816.084797:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828816.276712:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828816.276712:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828816.596740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828817.108750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828817.108812:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828817.300786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828817.300787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828817.620754:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828818.132810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828818.132942:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828818.324747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828818.324763:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828818.324780:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828818.324784:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828818.324794:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1824 00000100:00100000:3.0:1787828818.324794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828818.324807:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828818.324814:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828818.324850:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517980288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828818.324876:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828818.324889:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828818.324890:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828818.324894:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1824 00000100:00080000:1.0:1787828818.324897:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828818.324900:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828818.324909:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828818.324919:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517980288:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828818.324924:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517980416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828818.324933:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828818.324935:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828818.324937:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828818.324940:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1824 00000100:00080000:1.0:1787828818.324942:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828818.324955:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828818.324958:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828818.324967:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517980544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828818.324978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828818.324980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828818.324982:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828818.324984:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1824 00000100:00080000:1.0:1787828818.324986:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828818.324989:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787828818.324993:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787828818.324995:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828818.325005:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517980672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828818.325012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828818.325015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828818.325016:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828818.325019:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1824 00000100:00080000:1.0:1787828818.325021:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828818.325023:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828818.325026:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828818.325034:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517980800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828818.325039:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff8b2e88077840 x1874672517980288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828818.325053:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828818.325065:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828818.325068:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828818.325072:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517980288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828818.325076:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1824 00000100:00080000:1.0:1787828818.325085:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828818.325088:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828818.325093:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787828818.325095:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828818.325103:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517980288:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787828818.325105:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828818.325130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517980928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828818.325139:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517980416:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828818.325145:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828818.325150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828818.325153:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828818.325158:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1824 00000100:00080000:1.0:1787828818.325164:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828818.325169:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787828818.325174:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787828818.325178:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828818.325203:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff8b2e88076580 x1874672517980416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828818.325207:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517981056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828818.325223:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517980416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828818.325234:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828818.325236:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828818.325239:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517980416:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828818.325245:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517980544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828818.325270:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828818.325277:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff8b2e88076d00 x1874672517980544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828818.325285:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517980544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828818.325290:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828818.325292:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828818.325295:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517980544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828818.325298:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517980672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828818.325307:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828818.325313:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff8b2e880770c0 x1874672517980672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828818.325325:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517980672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828818.325348:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828818.325350:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828818.325353:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517980672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828818.325357:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517980800:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828818.325371:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828818.325388:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff8b2e88074000 x1874672517980800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828818.325395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517980800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828818.325401:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828818.325403:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828818.325405:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517980800:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828818.325417:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517980928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828818.325427:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828818.325433:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff8b2e880761c0 x1874672517980928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828818.325441:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517980928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828818.325450:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828818.325452:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828818.325454:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517980928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828818.325457:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517981056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828818.325467:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828818.325472:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828818/real 1787828818] req@ffff8b2e88075e00 x1874672517981056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828818.325479:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517981056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828843 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828818.325485:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828818.325486:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828818.325488:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517981056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828818.644793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828819.156734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828819.156785:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828819.348762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828819.348764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828819.668742:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828820.180765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828820.180826:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828820.372720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828820.372727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828820.692799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828821.204771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828821.204852:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828821.396770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828821.396781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828821.716736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828822.228749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828822.228861:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828822.420741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828822.420741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828822.740781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828823.252789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828823.252873:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828823.444768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828823.444778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828823.444783:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828823.444790:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828823.444796:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828823.444808:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1829 00000100:00080000:3.0:1787828823.444821:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.444830:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828823.444873:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672517981184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828823.444903:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828823.444907:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828823.444909:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828823.444913:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1829 00000100:00080000:3.0:1787828823.444917:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.444920:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828823.444930:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828823.444942:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517981184:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828823.444948:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672517981312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828823.444965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828823.444970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828823.444972:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828823.444978:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1829 00000100:00080000:3.0:1787828823.444983:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.444990:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828823.444997:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828823.445010:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672517981440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828823.445022:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828823.445022:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828823.445033:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828823.445035:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828823.445041:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1829 00000100:00080000:3.0:1787828823.445048:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.445052:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828823.445055:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828823.445064:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672517981568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828823.445072:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828823.445075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828823.445077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828823.445083:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff8b2e8803b480 x1874672517981184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828823.445085:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1829 00000100:00080000:3.0:1787828823.445088:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.445092:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828823.445103:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828823.445121:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672517981696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828823.445142:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828823.445149:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517981184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828823.445149:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828823.445157:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828823.445161:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1829 00000100:00080000:3.0:1787828823.445167:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.445174:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828823.445182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828823.445201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517981824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828823.445202:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828823.445220:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787828823.445223:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828823.445230:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517981184:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828823.445231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828823.445233:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828823.445246:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1829 00000100:00080000:3.0:1787828823.445250:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828823.445253:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828823.445257:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828823.445268:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517981952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828823.445273:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517981312:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828823.445319:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828823.445331:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff8b2e880392c0 x1874672517981312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828823.445341:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517981312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828823.445354:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828823.445356:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828823.445359:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517981312:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828823.445364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517981440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828823.445380:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828823.445387:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff8b2e88039e00 x1874672517981440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828823.445395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517981440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828823.445404:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828823.445407:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828823.445409:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517981440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828823.445413:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517981568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828823.445423:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828823.445428:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff8b2e8803a1c0 x1874672517981568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828823.445435:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517981568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828823.445441:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828823.445443:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828823.445445:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517981568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828823.445448:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517981696:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828823.445458:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828823.445464:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff8b2e88039680 x1874672517981696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828823.445471:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517981696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828823.445477:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828823.445478:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828823.445481:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517981696:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828823.445485:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517981824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828823.445495:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828823.445500:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff8b2e880383c0 x1874672517981824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828823.445507:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517981824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828823.445512:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828823.445514:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828823.445516:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517981824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828823.445529:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517981952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828823.445540:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828823.445546:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828823/real 1787828823] req@ffff8b2e8803a580 x1874672517981952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828823.445554:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517981952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828848 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828823.445559:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828823.445560:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828823.445562:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517981952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828823.764741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828824.276759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828824.276797:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828824.468720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828824.468720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828824.788777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828825.300729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828825.300794:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828825.492759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828825.492759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828825.812775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828826.324725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828826.324793:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828826.516767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828826.516774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828826.836719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828827.348740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828827.348777:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828827.540716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828827.540721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828827.860802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828828.372726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828828.372809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828828.564745:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828828.564755:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787828828.564760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828828.564760:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828828.564761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828828.564775:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1834 00000100:00080000:3.0:1787828828.564785:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828828.564793:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828828.564825:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672517982080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828828.564845:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828828.564848:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828828.564849:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828828.564852:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1834 00000100:00080000:3.0:1787828828.564854:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828828.564857:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828828.564867:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828828.564875:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517982080:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828828.564882:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672517982208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828828.564895:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828828.564899:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828828.564901:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828828.564905:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1834 00000100:00080000:3.0:1787828828.564910:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828828.564914:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828828.564921:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828828.564937:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672517982336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828828.564938:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828828.564953:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828828.564957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828828.564959:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828828.564964:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1834 00000100:00080000:3.0:1787828828.564970:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828828.564980:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff8b2e8803b840 x1874672517982080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828828.564985:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828828.564996:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828828.564999:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517982080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828828.565009:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828828.565014:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828828.565014:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672517982464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828828.565018:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517982080:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828828.565034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828828.565039:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828828.565041:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828828.565043:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517982208:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787828828.565050:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1834 00000100:00080000:3.0:1787828828.565055:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828828.565059:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787828828.565064:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:3.0:1787828828.565068:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828828.565076:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff8b2e88038f00 x1874672517982208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828828.565091:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517982208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828828.565091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672517982592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828828.565112:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828828.565117:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828828.565122:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517982208:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787828828.565125:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828828.565129:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517982336:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787828828.565134:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828828.565139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828828.565146:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1834 00000800:00000100:0.0:1787828828.565151:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828828.565155:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828828.565161:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff8b2e8803b0c0 x1874672517982336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828828.565164:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828828.565177:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828828.565179:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517982336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828828.565188:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828828.565191:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828828.565196:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517982336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828828.565197:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517982720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828828.565201:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517982464:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787828828.565216:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828828.565222:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828828.565223:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828828.565228:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828828.565232:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff8b2e88039a40 x1874672517982464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828828.565237:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1834 00000100:00080000:3.0:1787828828.565247:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828828.565249:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517982464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828828.565267:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828828.565277:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828828.565279:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828828.565282:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828828.565287:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517982464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828828.565300:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517982848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828828.565304:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517982592:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828828.565344:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828828.565355:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff8b2e8803b840 x1874672517982592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828828.565369:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517982592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828828.565375:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828828.565377:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828828.565379:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517982592:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828828.565382:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517982720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828828.565393:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828828.565399:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff8b2e8803a580 x1874672517982720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828828.565406:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517982720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828828.565412:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828828.565414:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828828.565416:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517982720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828828.565424:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517982848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828828.565433:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828828.565439:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828828/real 1787828828] req@ffff8b2e880383c0 x1874672517982848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828828.565446:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517982848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828853 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828828.565456:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828828.565459:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828828.565461:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517982848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828828.884732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828829.396759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828829.396819:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828829.588771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828829.588771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828829.908824:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828830.420872:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828830.420894:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828830.612786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828830.612786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828830.932753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828831.444740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828831.444807:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828831.636765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828831.636788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828831.956783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828832.468809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828832.468863:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828832.660797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828832.660797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828832.980777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828833.492711:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828833.492791:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828833.684711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828833.684719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828833.684782:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828833.684788:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828833.684791:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828833.684799:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1840 00000100:00080000:2.0:1787828833.684809:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828833.684815:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828833.684847:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517982976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828833.684867:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828833.684869:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828833.684871:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828833.684874:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1840 00000100:00080000:2.0:1787828833.684876:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828833.684878:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828833.684887:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828833.684895:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517983104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828833.684903:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517982976:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787828833.684905:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828833.684910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828833.684911:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828833.684913:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1840 00000100:00080000:2.0:1787828833.684915:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828833.684917:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828833.684921:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828833.684929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517983232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828833.684936:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828833.684938:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828833.684939:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828833.684942:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1840 00000100:00080000:2.0:1787828833.684944:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828833.684946:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828833.684949:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828833.684956:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517983360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828833.684961:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828833.684967:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828833.684970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828833.684971:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828833.684975:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1840 00000100:00080000:2.0:1787828833.684978:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828833.684981:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828833.684984:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828833.684991:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517983488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828833.684998:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828833.685000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828833.685003:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828833.685004:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828833] req@ffff8b2e8801da40 x1874672517982976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828833.685009:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1840 00000100:00080000:2.0:1787828833.685013:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828833.685019:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828833.685023:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828833.685028:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672517982976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828833.685033:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517983616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828833.685048:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828833.685049:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828833.685052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828833.685056:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828833.685060:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787828833.685061:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1840 00000100:00080000:2.0:1787828833.685065:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828833.685067:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828833.685069:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828833.685075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517983744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828833.685080:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517982976:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828833.685107:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517983104:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828833.685126:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828833.685153:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828833] req@ffff8b2e8801fc00 x1874672517983104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828833.685163:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517983104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828833.685170:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828833.685173:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828833.685175:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517983104:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828833.685180:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517983232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828833.685193:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828833.685200:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828833] req@ffff8b2e8801f480 x1874672517983232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828833.685208:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517983232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828833.685214:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828833.685216:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828833.685218:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517983232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828833.685239:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517983360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828833.685270:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828833.685277:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828833] req@ffff8b2e8801cf00 x1874672517983360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828833.685284:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672517983360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828833.685290:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828833.685292:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828833.685294:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517983360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828833.685298:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517983488:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828833.685327:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828833.685341:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828833] req@ffff8b2e8801c000 x1874672517983488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828833.685351:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672517983488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828833.685357:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828833.685359:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828833.685361:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517983488:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828833.685366:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517983616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828833.685377:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828833.685384:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828833] req@ffff8b2e8801e580 x1874672517983616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828833.685393:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517983616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828833.685399:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828833.685401:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828833.685413:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517983616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828833.685421:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517983744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828833.685435:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828833.685449:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828833/real 1787828833] req@ffff8b2e8801e940 x1874672517983744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828833.685470:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517983744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828858 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828833.685477:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828833.685479:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828833.685482:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517983744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828834.004795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828834.516764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828834.516867:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828834.708857:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828834.708858:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828835.028759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828835.540739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828835.540828:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828835.732794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828835.732909:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828836.052764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828836.564744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828836.564881:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828836.756794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828836.756819:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828837.076736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828837.588725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828837.588809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828837.780773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828837.780773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828838.100740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828838.612771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828838.612840:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828838.804734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828838.804734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828838.804786:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828838.804795:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828838.804801:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828838.804814:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1845 00000100:00080000:1.0:1787828838.804837:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828838.804848:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828838.804915:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672517983872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828838.804960:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828838.804963:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828838.804965:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828838.804969:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1845 00000100:00080000:1.0:1787828838.804972:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828838.804976:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828838.804986:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828838.805016:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672517984000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828838.805020:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517983872:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828838.805037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828838.805043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828838.805045:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828838.805050:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1845 00000100:00080000:1.0:1787828838.805055:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828838.805076:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828838.805082:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828838.805098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672517984128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828838.805112:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828838.805112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828838.805118:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828838.805121:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828838.805145:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1845 00000100:00080000:1.0:1787828838.805160:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828838.805166:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828838.805173:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828838.805181:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff8b2e88074b40 x1874672517983872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828838.805210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672517984256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828838.805221:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672517983872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828838.805249:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828838.805253:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828838.805255:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828838.805258:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828838.805260:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828838.805264:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517983872:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828838.805270:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1845 00000100:00080000:1.0:1787828838.805276:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828838.805281:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828838.805287:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517984000:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787828838.805289:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828838.805307:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672517984384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828838.805316:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828838.805327:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828838.805331:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828838.805332:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff8b2e88075e00 x1874672517984000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828838.805334:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828838.805345:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1845 00000100:00080000:1.0:1787828838.805353:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828838.805357:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672517984000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828838.805362:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828838.805389:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828838.805392:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828838.805399:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828838.805404:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517984000:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828838.805408:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672517984512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828838.805415:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517984128:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787828838.805427:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828838.805448:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828838.805450:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828838.805455:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1845 00000100:00080000:1.0:1787828838.805459:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828838.805464:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828838.805470:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828838.805478:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828838.805485:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672517984640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828838.805495:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff8b2e880761c0 x1874672517984128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828838.805509:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672517984128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828838.805518:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828838.805522:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828838.805525:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517984128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828838.805531:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517984256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828838.805551:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828838.805561:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff8b2e88074000 x1874672517984256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828838.805572:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672517984256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828838.805581:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828838.805585:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828838.805588:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517984256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828838.805604:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517984384:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828838.805624:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828838.805636:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff8b2e880770c0 x1874672517984384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828838.805672:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672517984384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828838.805680:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828838.805682:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828838.805685:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517984384:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828838.805690:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517984512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828838.805705:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828838.805713:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff8b2e88076d00 x1874672517984512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828838.805721:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672517984512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828838.805726:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828838.805728:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828838.805731:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517984512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828838.805735:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517984640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828838.805745:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828838.805750:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828838/real 1787828838] req@ffff8b2e88076580 x1874672517984640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828838.805757:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672517984640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828863 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828838.805763:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828838.805765:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828838.805766:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517984640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828839.124733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828839.636722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828839.636860:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828839.828775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828839.828775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828840.148737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828840.660766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828840.660869:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828840.852767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828840.852790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828841.172788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828841.684809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828841.684861:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828841.876746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828841.876746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828842.196719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828842.708790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828842.708808:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828842.900726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828842.900736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828843.220822:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828843.732799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828843.732859:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787828843.924781:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828843.924791:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787828843.924795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828843.924796:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828843.924810:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1850 00000100:00100000:2.0:1787828843.924817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828843.924821:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828843.924832:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828843.924873:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672517984768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828843.924897:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828843.924900:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828843.924901:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828843.924906:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1850 00000100:00080000:3.0:1787828843.924910:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828843.924913:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828843.924922:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828843.924928:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517984768:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828843.924935:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672517984896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828843.924943:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828843.924946:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828843.924948:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828843.924950:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1850 00000100:00080000:3.0:1787828843.924953:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828843.924955:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828843.924958:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828843.924969:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672517985024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828843.924975:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828843.924978:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828843.924979:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828843.924982:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1850 00000100:00080000:3.0:1787828843.924984:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828843.924986:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828843.924989:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828843.924998:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828843.924999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672517985152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828843.925008:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828843.925011:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828843.925012:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828843.925015:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1850 00000100:00080000:3.0:1787828843.925017:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828843.925020:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828843.925023:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828843.925029:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672517985280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828843.925035:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828843.925037:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828843.925039:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828843.925042:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1850 00000100:00080000:3.0:1787828843.925044:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828843.925047:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828843.925050:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828843.925055:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff8b2e88039680 x1874672517984768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828843.925060:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517985408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828843.925092:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828843.925094:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828843.925095:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828843.925098:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1850 00000100:00080000:3.0:1787828843.925101:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828843.925105:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517984768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828843.925107:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828843.925113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828843.925123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517985536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828843.925126:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828843.925136:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828843.925143:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517984768:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828843.925180:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517984896:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828843.925207:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828843.925217:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff8b2e8803a1c0 x1874672517984896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828843.925226:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517984896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828843.925232:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828843.925235:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828843.925237:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517984896:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828843.925241:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517985024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828843.925253:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828843.925259:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff8b2e88039e00 x1874672517985024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828843.925266:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517985024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828843.925271:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828843.925273:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828843.925275:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517985024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828843.925279:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517985152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828843.925288:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828843.925294:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff8b2e880392c0 x1874672517985152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828843.925301:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517985152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828843.925307:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828843.925308:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828843.925310:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517985152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828843.925313:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517985280:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828843.925323:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828843.925329:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff8b2e8803b480 x1874672517985280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828843.925336:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517985280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828843.925341:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828843.925342:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828843.925344:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517985280:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828843.925348:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517985408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828843.925357:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828843.925362:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff8b2e880383c0 x1874672517985408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828843.925369:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517985408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828843.925375:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828843.925376:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828843.925378:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517985408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828843.925382:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517985536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828843.925391:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828843.925396:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828843/real 1787828843] req@ffff8b2e8803a580 x1874672517985536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828843.925403:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517985536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828868 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828843.925408:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828843.925410:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828843.925411:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517985536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828844.244721:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828844.756738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828844.756770:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828844.948766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828844.948846:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828845.268799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828845.780751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828845.780834:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828845.972740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828845.972748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828846.292817:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828846.804781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828846.804809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828846.996731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828846.996736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828847.316784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828847.828746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828847.828811:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828848.020748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828848.020759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828848.340732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828848.852776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828848.852859:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828849.044753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828849.044754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828849.044779:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828849.044792:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828849.044798:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828849.044810:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1855 00000100:00080000:2.0:1787828849.044824:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828849.044833:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828849.044882:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517985664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828849.044913:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828849.044918:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828849.044920:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828849.044925:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1855 00000100:00080000:2.0:1787828849.044930:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828849.044936:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828849.044949:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828849.044961:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517985664:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828849.044970:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517985792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828849.044984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828849.044987:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828849.044990:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828849.044994:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1855 00000100:00080000:2.0:1787828849.045000:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828849.045005:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828849.045012:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1787828849.045026:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787828849.045029:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517985920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828849.045044:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828849.045057:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828849.045059:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828849.045065:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1855 00000100:00080000:2.0:1787828849.045070:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828849.045085:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828849.045092:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828849.045096:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828849/real 1787828849] req@ffff8b2e8801e940 x1874672517985664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828849.045110:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517986048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828849.045139:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517985664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828849.045139:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828849.045155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828849.045170:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828849.045187:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1855 00000100:00080000:0.0:1787828849.045193:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828849.045194:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828849.045199:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787828849.045206:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828849.045207:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517985664:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787828849.045222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828849.045245:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517986176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828849.045250:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517985792:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828849.045274:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828849.045279:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828849.045282:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828849.045294:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1855 00000800:00000100:0.0:1787828849.045295:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828849.045307:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828849.045316:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828849/real 1787828849] req@ffff8b2e8801e580 x1874672517985792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828849.045316:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828849.045331:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828849.045337:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517985792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828849.045347:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828849.045351:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787828849.045356:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517986304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828849.045358:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517985792:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828849.045375:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517985920:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787828849.045394:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828849.045399:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828849.045403:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828849.045404:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828849.045417:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1855 00000100:00100000:0.0:1787828849.045421:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828849/real 1787828849] req@ffff8b2e8801c000 x1874672517985920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828849.045425:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828849.045436:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828849.045451:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828849.045459:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672517985920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828849.045472:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517986432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828849.045546:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828849.045552:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828849.045556:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517985920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828849.045565:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517986048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828849.045601:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828849.045615:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828849/real 1787828849] req@ffff8b2e8801cf00 x1874672517986048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828849.045629:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672517986048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828849.045638:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828849.045641:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828849.045668:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517986048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828849.045687:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517986176:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828849.045708:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828849.045717:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828849/real 1787828849] req@ffff8b2e8801f480 x1874672517986176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828849.045725:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517986176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828849.045732:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828849.045734:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828849.045736:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517986176:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828849.045741:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517986304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828849.045756:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828849.045767:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828849/real 1787828849] req@ffff8b2e8801fc00 x1874672517986304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828849.045779:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517986304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828849.045788:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828849.045792:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828849.045796:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517986304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828849.045803:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517986432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828849.045820:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828849.045831:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828849/real 1787828849] req@ffff8b2e8801da40 x1874672517986432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828849.045843:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672517986432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828874 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828849.045851:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828849.045855:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828849.045858:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517986432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828849.364769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828849.876743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828849.876852:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828850.068769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828850.068769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828850.388762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828850.900783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828850.900848:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828851.092762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828851.092767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828851.412783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828851.924692:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828851.924796:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828852.116754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828852.116765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828852.436721:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828852.948750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828852.948791:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828853.140750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828853.140750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828853.460770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828853.972762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828853.972796:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828854.164733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828854.164737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828854.164741:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828854.164748:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828854.164752:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828854.164760:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1860 00000100:00080000:1.0:1787828854.164771:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828854.164778:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828854.164815:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672517986560/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828854.164837:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828854.164840:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828854.164842:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828854.164845:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1860 00000100:00080000:1.0:1787828854.164848:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828854.164851:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828854.164860:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828854.164871:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517986560:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828854.164872:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672517986688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828854.164885:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828854.164887:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828854.164889:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828854.164892:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1860 00000100:00080000:1.0:1787828854.164895:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828854.164897:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828854.164901:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828854.164910:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672517986816/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828854.164916:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828854.164918:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828854.164920:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828854.164922:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1860 00000100:00080000:1.0:1787828854.164924:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828854.164926:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828854.164929:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828854.164933:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787828854.164946:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672517986944/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828854.164957:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828854.164961:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828854.164963:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828854.164969:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1860 00000100:00080000:1.0:1787828854.164973:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828854.164974:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828854/real 1787828854] req@ffff8b2e88077840 x1874672517986560/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828854.164984:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828854.164992:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828854.164996:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672517986560/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828854.165007:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828854.165007:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672517987072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828854.165015:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828854.165022:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517986560:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828854.165024:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828854.165029:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828854.165031:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828854.165034:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1860 00000100:00080000:1.0:1787828854.165038:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828854.165041:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828854.165044:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828854.165046:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517986688:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828854.165054:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672517987200/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828854.165063:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828854.165067:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787828854.165068:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828854.165069:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828854.165075:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1860 00000100:00080000:1.0:1787828854.165094:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828854.165097:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828854/real 1787828854] req@ffff8b2e88074780 x1874672517986688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828854.165102:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828854.165110:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828854.165115:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672517986688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828854.165123:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672517987328/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828854.165125:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828854.165133:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828854.165140:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517986688:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828854.165147:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517986816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828854.165162:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828854.165170:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828854/real 1787828854] req@ffff8b2e88076940 x1874672517986816/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828854.165177:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672517986816/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828854.165187:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828854.165189:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828854.165191:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517986816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828854.165194:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517986944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828854.165204:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828854.165210:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828854/real 1787828854] req@ffff8b2e88074f00 x1874672517986944/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828854.165217:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672517986944/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828854.165222:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828854.165224:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828854.165226:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517986944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828854.165229:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517987072:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828854.165239:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828854.165244:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828854/real 1787828854] req@ffff8b2e88075680 x1874672517987072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828854.165252:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672517987072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828854.165257:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828854.165258:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828854.165261:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517987072:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828854.165273:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517987200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828854.165282:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828854.165288:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828854/real 1787828854] req@ffff8b2e88075a40 x1874672517987200/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828854.165295:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672517987200/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828854.165300:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828854.165302:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828854.165304:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517987200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828854.165308:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517987328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828854.165316:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828854.165322:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828854/real 1787828854] req@ffff8b2e88077480 x1874672517987328/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828854.165329:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672517987328/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828879 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828854.165334:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828854.165335:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828854.165337:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517987328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828854.484733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828854.996737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828854.996804:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828855.188782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828855.188801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828855.508878:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828856.020766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828856.020872:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828856.212741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828856.212741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828856.532722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828857.044727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828857.044772:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828857.236739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828857.236765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828857.556747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828858.068790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828858.068921:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828858.260734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828858.260792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828858.580769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828859.092766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828859.092859:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828859.284758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828859.284761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828859.284805:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828859.284812:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828859.284818:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828859.284828:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1865 00000100:00080000:0.0:1787828859.284842:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828859.284850:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828859.284896:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517987456/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828859.284933:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828859.284936:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828859.284938:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828859.284942:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1865 00000100:00080000:0.0:1787828859.284946:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828859.284950:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828859.284962:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828859.284974:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517987456:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828859.284979:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517987584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828859.284990:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828859.284994:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828859.284995:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828859.284999:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1865 00000100:00080000:0.0:1787828859.285003:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828859.285006:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828859.285009:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828859.285019:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517987712/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828859.285028:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828859.285031:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828859.285033:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828859.285036:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1865 00000100:00080000:0.0:1787828859.285039:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828859.285041:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828859.285063:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828859.285074:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517987840/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828859.285081:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828859.285084:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828859.285086:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828859.285090:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1865 00000100:00080000:0.0:1787828859.285093:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828859.285096:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828859.285100:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828859.285111:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517987968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828859.285119:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828859.285122:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828859.285123:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828859.285129:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517987584:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828859.285131:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1865 00000100:00080000:0.0:1787828859.285138:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828859.285141:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828859.285145:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828859.285158:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517988096/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828859.285168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828859.285171:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517987712:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787828859.285173:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828859.285178:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828859.285184:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1865 00000100:00080000:0.0:1787828859.285188:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828859.285192:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828859.285198:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828859.285202:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517987840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828859.285212:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517988224/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828859.285232:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517987968:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828859.285254:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517988096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828859.285271:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517988224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828859.604828:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828860.116795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828860.116855:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828860.308756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828860.308760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828860.628739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828861.140798:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828861.140866:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828861.332748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828861.332749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828861.652782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828862.164743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828862.164845:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828862.356758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828862.356801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:3.0:1787828862.357095:0:2336:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:3.0:1787828862.357150:0:2336:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787828862.357166:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787828862.357206:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787828862.357215:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787828862.357224:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787828862.357250:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787828862.357260:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787828862.357271:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787828862.357285:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828862] req@ffff8b2e88040f00 x1874672517988096/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828862.357333:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517988096/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828862.357345:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828862.357351:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828862.357356:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517988096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828862.357374:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828862] req@ffff8b2e88041a40 x1874672517988224/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828862.357387:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517988224/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828862.357396:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828862.357399:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828862.357403:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517988224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828862.357408:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828862] req@ffff8b2e88040000 x1874672517987584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828862.357436:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517987584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828862.357444:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828862.357449:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828862.357452:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517987584:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828862.357464:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828862] req@ffff8b2e880430c0 x1874672517987712/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828862.357477:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517987712/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828862.357484:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828862.357488:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828862.357491:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517987712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828862.357498:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828862] req@ffff8b2e88043480 x1874672517987840/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828862.357510:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517987840/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828862.357517:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828862.357520:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828862.357524:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517987840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828862.357530:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828862] req@ffff8b2e88042940 x1874672517987968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828862.357541:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517987968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828862.357549:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828862.357557:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828862.357561:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517987968:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828862.357569:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828859/real 1787828862] req@ffff8b2e880403c0 x1874672517987456/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828862.357582:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672517987456/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828884 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828862.357593:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828862.357597:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828862.357601:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517987456:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828862.676755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828863.188728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828863.188816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828863.380721:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828863.380734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828863.700735:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828864.212781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828864.212941:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787828864.404778:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787828864.404793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828864.404794:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787828864.404794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828864.404806:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828864.404821:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1870 00000100:00080000:0.0:1787828864.404837:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828864.404845:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828864.404888:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672517988352/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828864.404921:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828864.404926:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828864.404927:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828864.404932:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1870 00000100:00080000:0.0:1787828864.404935:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828864.404940:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828864.404950:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828864.404962:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517988352:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828864.404967:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517988480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828864.404982:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828864.404986:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828864.404988:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828864.404992:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1870 00000100:00080000:0.0:1787828864.404995:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828864.405000:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828864.405020:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828864.405034:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517988608/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828864.405045:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828864.405048:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828864.405053:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828864.405056:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828864.405061:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1870 00000100:00080000:0.0:1787828864.405071:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828864.405076:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828864.405081:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828864.405098:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff8b2e880403c0 x1874672517988352/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828864.405099:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517988736/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828864.405112:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828864.405121:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828864.405125:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828864.405134:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1870 00000100:00080000:0.0:1787828864.405139:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828864.405145:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672517988352/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828864.405146:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828864.405156:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828864.405165:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828864.405173:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828864.405175:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517988864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828864.405182:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517988352:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828864.405194:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828864.405199:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828864.405202:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828864.405207:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1870 00000100:00080000:0.0:1787828864.405211:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828864.405216:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828864.405220:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517988480:192.168.203.101@tcp:38:kworker.0 00000008:00100000:0.0:1787828864.405224:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828864.405242:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517988992/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828864.405252:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828864.405258:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787828864.405259:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828864.405262:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828864.405270:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1870 00000100:00080000:0.0:1787828864.405275:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828864.405280:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff8b2e88042940 x1874672517988480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828864.405282:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828864.405290:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828864.405307:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517988480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828864.405311:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517989120/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828864.405329:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828864.405337:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828864.405342:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517988480:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828864.405350:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517988608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828864.405378:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828864.405392:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff8b2e88043480 x1874672517988608/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828864.405405:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517988608/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828864.405421:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828864.405425:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828864.405429:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517988608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828864.405436:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517988736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828864.405460:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828864.405474:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff8b2e880430c0 x1874672517988736/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828864.405487:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517988736/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828864.405496:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828864.405500:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828864.405503:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517988736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828864.405509:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517988864:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828864.405530:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828864.405542:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff8b2e88040000 x1874672517988864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828864.405556:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517988864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828864.405565:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828864.405568:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828864.405572:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517988864:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828864.405602:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517988992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828864.405626:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828864.405639:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff8b2e88041a40 x1874672517988992/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828864.405811:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517988992/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828864.405820:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828864.405823:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828864.405827:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517988992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828864.405834:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517989120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828864.405857:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828864.405870:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828864/real 1787828864] req@ffff8b2e88040f00 x1874672517989120/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828864.405882:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517989120/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828889 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828864.405891:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828864.405894:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828864.405898:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517989120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828864.724830:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828865.236714:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828865.236875:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828865.428769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828865.428769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828865.748761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828866.260831:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828866.260904:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828866.452751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828866.452751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828866.772723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828867.284720:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828867.284807:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828867.476787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828867.476819:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828867.796725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828868.308729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828868.308805:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828868.500717:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828868.500746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828868.820712:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828869.332720:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828869.332788:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828869.524741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828869.524741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828869.524758:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828869.524764:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828869.524769:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828869.524779:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1875 00000100:00080000:0.0:1787828869.524793:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828869.524802:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828869.524850:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517989248/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828869.524881:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828869.524885:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828869.524886:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828869.524890:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1875 00000100:00080000:0.0:1787828869.524895:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828869.524900:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828869.524910:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517989248:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787828869.524911:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828869.524927:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517989376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828869.524935:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828869.524938:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828869.524939:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828869.524942:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1875 00000100:00080000:0.0:1787828869.524945:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828869.524947:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828869.524950:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828869.524958:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517989504/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828869.524964:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828869.524966:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828869.524968:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828869.524985:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1875 00000100:00080000:0.0:1787828869.524990:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828869.524993:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828869.524998:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828869.525008:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517989632/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828869.525013:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828869.525021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828869.525030:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828869.525033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828869.525039:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1875 00000100:00080000:0.0:1787828869.525044:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828869.525049:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828869.525054:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828869.525065:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff8b2e880412c0 x1874672517989248/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828869.525071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517989760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828869.525089:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828869.525091:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672517989248/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828869.525096:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828869.525100:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828869.525110:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1875 00000100:00080000:3.0:1787828869.525111:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828869.525114:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828869.525118:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787828869.525122:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828869.525126:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517989248:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787828869.525129:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828869.525141:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517989888/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828869.525151:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517989376:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828869.525160:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828869.525164:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828869.525166:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828869.525169:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1875 00000100:00080000:0.0:1787828869.525172:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828869.525174:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828869.525177:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828869.525188:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828869.525189:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517990016/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828869.525202:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff8b2e88043840 x1874672517989376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828869.525214:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517989376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828869.525221:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828869.525223:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828869.525226:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517989376:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828869.525230:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517989504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828869.525243:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828869.525250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff8b2e88040780 x1874672517989504/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828869.525258:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517989504/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828869.525264:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828869.525266:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828869.525267:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517989504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828869.525271:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517989632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828869.525281:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828869.525287:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff8b2e88042d00 x1874672517989632/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828869.525294:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517989632/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828869.525300:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828869.525302:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828869.525303:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517989632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828869.525307:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517989760:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828869.525317:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828869.525323:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff8b2e880421c0 x1874672517989760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828869.525330:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517989760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828869.525340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828869.525342:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828869.525345:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517989760:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828869.525364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517989888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828869.525376:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828869.525383:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff8b2e88040b40 x1874672517989888/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828869.525391:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517989888/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828869.525396:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828869.525399:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828869.525401:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517989888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828869.525404:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517990016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828869.525413:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828869.525419:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828869/real 1787828869] req@ffff8b2e88041e00 x1874672517990016/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828869.525426:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517990016/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828894 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828869.525431:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828869.525433:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828869.525436:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517990016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828869.844701:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828870.356767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828870.356809:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828870.548729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828870.548799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828870.868706:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828871.380750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828871.380848:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828871.572797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828871.572797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828871.892729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828872.404782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828872.404810:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828872.596760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828872.596776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828872.916779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828873.428758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828873.428822:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828873.620784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828873.620784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828873.940708:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828874.452746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828874.452801:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828874.644734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828874.644734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828874.644749:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828874.644755:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828874.644760:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828874.644772:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1881 00000100:00080000:0.0:1787828874.644784:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828874.644793:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828874.644833:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672517990144/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828874.644857:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828874.644861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828874.644863:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828874.644869:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1881 00000100:00080000:0.0:1787828874.644873:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828874.644878:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828874.644890:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828874.644898:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517990144:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828874.644908:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672517990272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828874.644919:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828874.644923:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828874.644925:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828874.644929:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1881 00000100:00080000:0.0:1787828874.644932:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828874.644950:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828874.644956:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828874.644983:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672517990400/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828874.644992:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828874.644994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828874.644998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828874.645000:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828874.645004:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1881 00000100:00080000:0.0:1787828874.645007:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828874.645011:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828874.645015:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828874.645025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672517990528/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828874.645034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828874.645038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787828874.645040:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff8b2e88042580 x1874672517990144/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828874.645041:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828874.645046:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1881 00000100:00080000:0.0:1787828874.645049:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828874.645053:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828874.645058:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828874.645059:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672517990144/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828874.645068:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672517990656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828874.645072:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828874.645077:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787828874.645078:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828874.645081:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517990144:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828874.645082:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828874.645084:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828874.645088:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1881 00000100:00080000:0.0:1787828874.645092:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828874.645095:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828874.645100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828874.645105:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517990272:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828874.645110:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672517990784/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828874.645120:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828874.645124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828874.645126:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828874.645130:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1881 00000800:00000100:3.0:1787828874.645130:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828874.645134:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828874.645138:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828874.645143:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828874.645144:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff8b2e88041e00 x1874672517990272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828874.645155:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672517990912/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828874.645160:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672517990272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828874.645171:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828874.645175:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828874.645196:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517990272:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828874.645204:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517990400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828874.645234:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828874.645250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff8b2e88040b40 x1874672517990400/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828874.645261:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672517990400/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828874.645269:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828874.645272:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828874.645275:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517990400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828874.645280:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517990528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828874.645297:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828874.645306:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff8b2e880421c0 x1874672517990528/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828874.645315:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672517990528/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828874.645323:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828874.645326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828874.645329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517990528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828874.645333:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517990656:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828874.645351:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828874.645380:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff8b2e88042d00 x1874672517990656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828874.645394:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672517990656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828874.645405:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828874.645408:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828874.645412:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517990656:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828874.645449:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517990784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828874.645470:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828874.645481:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff8b2e88040780 x1874672517990784/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828874.645492:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672517990784/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828874.645515:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828874.645517:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828874.645520:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517990784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828874.645524:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517990912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828874.645537:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828874.645544:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828874/real 1787828874] req@ffff8b2e88043840 x1874672517990912/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828874.645553:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672517990912/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828899 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828874.645558:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828874.645576:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828874.645578:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517990912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828874.964755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828875.476763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828875.476793:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828875.668760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828875.668844:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828875.988716:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828876.500711:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828876.500796:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828876.692737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828876.692745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828877.012788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828877.524769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828877.524863:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828877.716781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828877.716782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828878.036784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828878.548805:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828878.548863:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828878.740709:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828878.740731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828879.060814:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828879.572782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828879.572866:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828879.764772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828879.764772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787828879.764781:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.764789:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828879.764794:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.764805:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1886 00000100:00080000:0.0:1787828879.764816:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.764824:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828879.764864:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672517991040/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828879.764893:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.764896:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828879.764897:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.764914:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1886 00000100:00080000:0.0:1787828879.764919:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.764923:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828879.764932:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828879.764935:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517991040:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828879.764947:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672517991168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828879.764956:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.764959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828879.764961:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.764964:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1886 00000100:00080000:0.0:1787828879.764969:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.764974:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828879.764979:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828879.764993:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672517991296/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828879.765003:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828879.765009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.765014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828879.765017:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.765022:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1886 00000100:00080000:0.0:1787828879.765028:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.765032:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828879.765037:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828879.765051:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672517991424/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828879.765062:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828879.765062:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff8b2e880412c0 x1874672517991040/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828879.765071:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828879.765075:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.765085:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1886 00000100:00080000:0.0:1787828879.765090:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.765094:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828879.765097:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672517991040/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787828879.765101:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828879.765113:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828879.765117:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828879.765122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672517991552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828879.765123:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517991040:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828879.765138:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828879.765143:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828879.765145:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828879.765149:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517991168:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828879.765152:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1886 00000100:00080000:0.0:1787828879.765158:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828879.765164:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828879.765169:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828879.765175:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828879.765187:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672517991680/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828879.765190:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff8b2e88040f00 x1874672517991168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828879.765216:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828879.765219:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672517991168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828879.765221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828879.765225:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828879.765235:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1886 00000100:00080000:3.0:1787828879.765237:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828879.765239:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828879.765243:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787828879.765246:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828879.765252:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517991168:192.168.203.101@tcp:38:kworker.0 00000008:00100000:0.0:1787828879.765253:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828879.765266:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517991296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828879.765269:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672517991808/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828879.765299:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828879.765312:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff8b2e88041a40 x1874672517991296/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828879.765321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672517991296/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828879.765340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828879.765342:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828879.765345:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517991296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828879.765350:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517991424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828879.765367:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828879.765379:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff8b2e88040000 x1874672517991424/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828879.765392:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672517991424/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828879.765402:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828879.765406:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828879.765410:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517991424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828879.765416:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517991552:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828879.765432:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828879.765441:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff8b2e880430c0 x1874672517991552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828879.765449:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672517991552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828879.765455:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828879.765457:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828879.765460:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517991552:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828879.765473:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517991680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828879.765484:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828879.765491:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff8b2e88043480 x1874672517991680/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828879.765499:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672517991680/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828879.765504:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828879.765506:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828879.765508:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517991680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828879.765512:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517991808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828879.765521:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828879.765529:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828879/real 1787828879] req@ffff8b2e88042940 x1874672517991808/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828879.765541:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672517991808/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828904 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828879.765549:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828879.765552:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828879.765556:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517991808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828880.084726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828880.596761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828880.596833:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828880.788721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828880.788749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828881.108736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828881.620817:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828881.620900:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828881.812771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828881.812782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828882.132835:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828882.644747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828882.644828:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828882.836730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828882.836730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828883.156733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828883.668758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828883.668849:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828883.860750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787828883.860757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828884.180792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828884.692728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828884.692792:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828884.884709:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828884.884751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828884.884754:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828884.884762:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828884.884766:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828884.884777:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1891 00000100:00080000:2.0:1787828884.884788:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828884.884796:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828884.884835:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517991936/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828884.884862:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828884.884879:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828884.884881:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828884.884887:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1891 00000100:00080000:2.0:1787828884.884892:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828884.884897:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828884.884900:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517991936:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787828884.884909:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828884.884929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517992064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828884.884941:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828884.884945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828884.884947:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828884.884950:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1891 00000100:00080000:2.0:1787828884.884953:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828884.884957:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828884.884960:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828884.884966:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828884.884977:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517992192/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828884.884984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828884.884986:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828884.884987:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828884.884990:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1891 00000100:00080000:2.0:1787828884.884993:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828884.884996:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828884.884999:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828884.885007:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff8b2e8801c780 x1874672517991936/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828884.885009:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517992320/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828884.885047:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828884.885051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787828884.885052:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517991936/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828884.885053:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828884.885061:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1891 00000100:00080000:2.0:1787828884.885066:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828884.885071:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787828884.885073:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828884.885075:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828884.885079:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828884.885084:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517991936:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828884.885088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517992448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828884.885096:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828884.885099:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828884.885100:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828884.885118:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1891 00000100:00080000:2.0:1787828884.885121:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828884.885125:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517992064:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828884.885126:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828884.885133:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828884.885143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517992576/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828884.885156:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828884.885157:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828884.885161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828884.885163:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828884.885167:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1891 00000100:00080000:2.0:1787828884.885170:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828884.885171:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff8b2e8801ed00 x1874672517992064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828884.885176:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828884.885183:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828884.885194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517992704/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828884.885199:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517992064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828884.885211:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828884.885214:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828884.885216:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517992064:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828884.885221:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517992192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828884.885235:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828884.885242:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff8b2e8801cb40 x1874672517992192/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828884.885250:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517992192/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828884.885255:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828884.885257:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828884.885259:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517992192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828884.885263:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517992320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828884.885278:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828884.885284:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff8b2e8801c3c0 x1874672517992320/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828884.885292:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517992320/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828884.885301:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828884.885312:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828884.885314:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517992320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828884.885318:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517992448:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828884.885329:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828884.885336:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff8b2e8801d680 x1874672517992448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828884.885344:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517992448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828884.885366:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828884.885369:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828884.885371:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517992448:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828884.885383:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517992576:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828884.885395:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828884.885401:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff8b2e8801de00 x1874672517992576/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828884.885409:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517992576/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828884.885415:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828884.885417:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828884.885419:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517992576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828884.885423:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517992704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828884.885432:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828884.885437:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828884/real 1787828884] req@ffff8b2e8801f0c0 x1874672517992704/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828884.885444:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517992704/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828909 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828884.885450:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828884.885451:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828884.885453:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517992704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828885.204768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828885.716749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828885.716819:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828885.908754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828885.908764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828886.228822:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828886.740730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828886.740801:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828886.932764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828886.932764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828887.252762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828887.764775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828887.764839:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828887.956758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828887.956758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828888.276793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828888.788752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828888.788831:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828888.980745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828888.980745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828889.300781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828889.812767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828889.812877:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828890.004765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828890.004789:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828890.004796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828890.004801:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828890.004811:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828890.004823:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1896 00000100:00080000:2.0:1787828890.004857:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828890.004867:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828890.004912:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517992832/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828890.004942:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828890.004946:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828890.004948:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828890.004952:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1896 00000100:00080000:2.0:1787828890.004956:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828890.004960:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828890.004972:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828890.004985:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517992960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828890.004997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828890.005002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828890.005007:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828890.005009:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517992832:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787828890.005012:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1896 00000100:00080000:2.0:1787828890.005029:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828890.005038:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828890.005044:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828890.005057:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517993088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828890.005067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828890.005071:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828890.005074:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828890.005077:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1896 00000100:00080000:2.0:1787828890.005083:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828890.005087:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828890.005091:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828890.005107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517993216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828890.005110:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828890.005124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828890.005129:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828890.005132:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828890.005136:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1896 00000100:00080000:2.0:1787828890.005141:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828890.005145:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828890.005150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828890.005160:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517993344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828890.005170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828890.005173:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828890.005175:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828890.005180:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1896 00000100:00080000:2.0:1787828890.005185:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828890.005191:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828890.005194:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828890/real 1787828890] req@ffff8b2e8801f0c0 x1874672517992832/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828890.005196:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828890.005213:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517993472/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828890.005231:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828890.005235:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828890.005237:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828890.005245:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1896 00000100:00100000:3.0:1787828890.005245:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517992832/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828890.005253:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828890.005260:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828890.005268:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828890.005285:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787828890.005286:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517993600/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828890.005301:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828890.005312:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517992832:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828890.005346:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517992960:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828890.005399:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828890.005415:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828890/real 1787828890] req@ffff8b2e8801de00 x1874672517992960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828890.005430:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517992960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828890.005441:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828890.005445:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828890.005449:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517992960:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828890.005457:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517993088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828890.005478:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828890.005491:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828890/real 1787828890] req@ffff8b2e8801d680 x1874672517993088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828890.005513:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517993088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828890.005522:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828890.005526:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828890.005529:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517993088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828890.005536:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517993216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828890.005554:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828890.005566:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828890/real 1787828890] req@ffff8b2e8801c3c0 x1874672517993216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828890.005580:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517993216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828890.005589:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828890.005593:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828890.005597:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517993216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828890.005604:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517993344:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828890.005628:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828890.005641:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828890/real 1787828890] req@ffff8b2e8801cb40 x1874672517993344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828890.005656:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517993344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828890.005700:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828890.005705:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828890.005709:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517993344:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828890.005717:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517993472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828890.005741:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828890.005756:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828890/real 1787828890] req@ffff8b2e8801ed00 x1874672517993472/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828890.005771:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517993472/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828890.005781:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828890.005785:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828890.005788:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517993472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828890.005796:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517993600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828890.005818:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828890.005850:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828890/real 1787828890] req@ffff8b2e8801c780 x1874672517993600/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828890.005866:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517993600/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828915 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828890.005876:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828890.005880:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828890.005884:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517993600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828890.324748:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828890.836778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828890.836911:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828891.028764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828891.028765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828891.348784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828891.860734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828891.860812:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828892.052768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828892.052773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828892.372785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828892.884753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828892.884824:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828893.076785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828893.076786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828893.396722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828893.908715:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828893.908789:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828894.100788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828894.100792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828894.420742:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828894.932756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828894.932779:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828895.124762:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828895.124764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828895.124764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828895.124774:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828895.124778:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828895.124789:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1901 00000100:00080000:2.0:1787828895.124816:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828895.124825:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828895.124874:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517993728/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828895.124897:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828895.124899:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828895.124900:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828895.124904:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1901 00000100:00080000:2.0:1787828895.124907:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828895.124910:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828895.124919:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828895.124928:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517993728:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828895.124930:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517993856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828895.124940:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828895.124943:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828895.124944:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828895.124948:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1901 00000100:00080000:2.0:1787828895.124950:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828895.124953:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828895.124956:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828895.124964:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517993984/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828895.124970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828895.124973:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828895.124975:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828895.124978:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1901 00000100:00080000:2.0:1787828895.124980:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828895.124982:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828895.124985:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828895.124988:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787828895.124997:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517994112/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828895.125004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828895.125006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828895.125007:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828895.125010:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1901 00000100:00080000:2.0:1787828895.125013:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828895.125015:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828895.125018:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828895.125025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517994240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828895.125032:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828895.125034:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828895/real 1787828895] req@ffff8b2e8801da40 x1874672517993728/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828895.125035:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828895.125049:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828895.125057:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1901 00000100:00080000:2.0:1787828895.125061:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828895.125064:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828895.125067:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828895.125068:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672517993728/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828895.125077:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517994368/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828895.125080:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828895.125088:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828895.125089:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787828895.125095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787828895.125096:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517993728:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787828895.125098:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828895.125107:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1901 00000100:00080000:2.0:1787828895.125118:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828895.125121:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828895.125124:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828895.125130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517994496/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828895.125135:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517993856:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828895.125155:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828895.125165:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828895/real 1787828895] req@ffff8b2e8801fc00 x1874672517993856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828895.125173:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517993856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828895.125179:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828895.125181:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828895.125184:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517993856:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828895.125187:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517993984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828895.125198:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828895.125204:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828895/real 1787828895] req@ffff8b2e8801f480 x1874672517993984/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828895.125212:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517993984/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828895.125217:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828895.125219:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828895.125220:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517993984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828895.125224:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517994112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828895.125233:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828895.125239:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828895/real 1787828895] req@ffff8b2e8801cf00 x1874672517994112/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828895.125250:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672517994112/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828895.125265:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828895.125268:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828895.125271:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517994112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828895.125278:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517994240:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828895.125298:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828895.125310:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828895/real 1787828895] req@ffff8b2e8801c000 x1874672517994240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828895.125318:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672517994240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828895.125324:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828895.125326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828895.125328:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517994240:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828895.125332:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517994368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828895.125346:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828895.125362:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828895/real 1787828895] req@ffff8b2e8801e580 x1874672517994368/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828895.125369:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517994368/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828895.125374:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828895.125376:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828895.125379:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517994368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828895.125392:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517994496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828895.125403:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828895.125409:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828895/real 1787828895] req@ffff8b2e8801e940 x1874672517994496/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828895.125416:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517994496/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828920 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828895.125422:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828895.125424:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828895.125426:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517994496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828895.444725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828895.956713:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828895.956782:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828896.148734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828896.148734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828896.468705:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828896.980728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828896.980806:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828897.172716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828897.172848:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828897.492733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787828898.004799:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828898.004844:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787828898.196762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828898.196811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828898.516709:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828899.028799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828899.028890:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828899.220846:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828899.220846:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828899.540716:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828900.052778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828900.052890:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828900.244721:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828900.244811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828900.244813:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.244824:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828900.244830:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.244844:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1906 00000100:00080000:2.0:1787828900.244861:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.244872:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828900.244922:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672517994624/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828900.244957:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.244962:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828900.244966:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.244976:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1906 00000100:00080000:2.0:1787828900.244981:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.244987:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828900.245008:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517994624:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787828900.245015:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828900.245038:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672517994752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828900.245051:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.245055:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828900.245057:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.245063:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1906 00000100:00080000:2.0:1787828900.245082:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.245088:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828900.245095:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787828900.245102:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787828900.245112:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672517994880/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828900.245126:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.245130:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828900.245133:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.245139:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1906 00000100:00080000:2.0:1787828900.245145:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.245150:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828900.245155:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828900] req@ffff8b2e8801e940 x1874672517994624/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828900.245159:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828900.245182:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672517994624/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828900.245190:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672517995008/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828900.245196:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828900.245205:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787828900.245214:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828900.245215:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517994624:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787828900.245232:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828900.245235:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.245242:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1906 00000100:00080000:2.0:1787828900.245248:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828900.245253:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517994752:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828900.245258:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828900.245268:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828900.245283:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787828900.245289:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672517995136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828900.245296:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828900] req@ffff8b2e8801e580 x1874672517994752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828900.245329:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828900.245329:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672517994752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828900.245351:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828900.245355:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828900.245356:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828900.245361:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828900.245373:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517994752:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828900.245374:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1906 00000100:00080000:2.0:1787828900.245388:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828900.245389:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517994880:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787828900.245396:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828900.245407:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787828900.245436:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787828900.245442:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672517995264/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828900.245455:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828900] req@ffff8b2e8801c000 x1874672517994880/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828900.245456:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828900.245471:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828900.245479:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828900.245494:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1906 00000100:00080000:2.0:1787828900.245499:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828900.245502:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828900.245510:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828900.245523:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672517994880/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828900.245524:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672517995392/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828900.245589:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828900.245597:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828900.245604:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517994880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828900.245611:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517995008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828900.245631:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828900.245640:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828900] req@ffff8b2e8801cf00 x1874672517995008/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828900.245670:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672517995008/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828900.245676:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828900.245679:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828900.245681:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517995008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828900.245694:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517995136:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828900.245708:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828900.245716:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828900] req@ffff8b2e8801f480 x1874672517995136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828900.245724:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672517995136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828900.245735:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828900.245737:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828900.245740:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517995136:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828900.245743:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517995264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828900.245754:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828900.245777:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828900] req@ffff8b2e8801fc00 x1874672517995264/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828900.245795:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672517995264/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828900.245801:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828900.245803:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828900.245805:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517995264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828900.245810:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517995392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828900.245823:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828900.245830:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828900/real 1787828900] req@ffff8b2e8801da40 x1874672517995392/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828900.245838:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672517995392/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828925 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828900.245854:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828900.245856:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828900.245859:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517995392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828900.564754:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828901.076788:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828901.076889:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828901.268734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828901.268789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828901.588777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828902.100836:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828902.100922:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828902.292729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828902.292806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828902.612727:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828903.124794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828903.124834:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828903.316774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828903.316778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828903.636777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828904.148725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828904.148862:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828904.340802:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828904.340804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828904.660782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828905.172771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828905.172843:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828905.364781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828905.364782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828905.364833:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.364844:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828905.364851:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828905.364865:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1911 00000100:00080000:3.0:1787828905.364876:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.364885:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828905.364927:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672517995520/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828905.364961:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.364967:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828905.364970:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828905.364978:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1911 00000100:00080000:3.0:1787828905.364985:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.365010:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828905.365018:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517995520:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787828905.365029:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828905.365056:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672517995648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828905.365071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.365076:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828905.365078:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828905.365082:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1911 00000100:00080000:3.0:1787828905.365087:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.365091:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828905.365097:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1787828905.365099:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828905.365118:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672517995776/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828905.365134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.365140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828905.365143:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828905.365151:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1911 00000100:00100000:0.0:1787828905.365157:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff8b2e8803b840 x1874672517995520/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828905.365162:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828905.365186:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517995520/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828905.365191:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828905.365207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828905.365215:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828905.365228:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828905.365230:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672517995904/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828905.365245:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517995520:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828905.365249:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.365260:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828905.365263:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828905.365270:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1911 00000100:00080000:3.0:1787828905.365275:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.365279:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828905.365285:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828905.365292:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517995648:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828905.365339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672517996032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828905.365356:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.365380:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:0.0:1787828905.365388:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828905.365389:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828905.365404:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1911 00000100:00080000:3.0:1787828905.365412:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828905.365418:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828905.365423:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff8b2e88039a40 x1874672517995648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828905.365431:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828905.365444:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517995648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828905.365473:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828905.365478:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517996160/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828905.365481:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828905.365518:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517995648:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828905.365533:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517995776:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787828905.365544:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828905.365551:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828905.365555:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828905.365564:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1911 00000800:00000100:0.0:1787828905.365565:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828905.365577:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828905.365584:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff8b2e8803b0c0 x1874672517995776/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828905.365589:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828905.365603:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828905.365610:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517995776/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828905.365621:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828905.365624:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517996288/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828905.365629:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828905.365657:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517995776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828905.365700:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517995904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828905.365768:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828905.365807:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff8b2e88038f00 x1874672517995904/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828905.365830:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517995904/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828905.365845:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828905.365851:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828905.365857:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517995904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828905.365884:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517996032:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828905.365910:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828905.365924:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff8b2e8803b840 x1874672517996032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828905.365955:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517996032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828905.365963:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828905.365966:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828905.365969:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517996032:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828905.365975:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517996160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828905.365990:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828905.365999:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff8b2e8803a580 x1874672517996160/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828905.366027:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517996160/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828905.366035:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828905.366038:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828905.366041:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517996160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828905.366047:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517996288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828905.366063:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828905.366073:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828905/real 1787828905] req@ffff8b2e880383c0 x1874672517996288/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828905.366084:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517996288/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828930 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828905.366093:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828905.366096:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828905.366099:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517996288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828905.684789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828906.196786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828906.196892:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828906.388737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828906.388751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828906.708729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828907.220790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828907.220835:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828907.412788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828907.412824:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828907.732789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828908.244789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828908.244870:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828908.436806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828908.436809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828908.756837:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828909.268800:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828909.268846:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828909.460708:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828909.460739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828909.780817:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828910.292761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828910.292820:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828910.484800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828910.484800:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.484812:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828910.484817:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.484827:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1916 00000100:00100000:1.0:1787828910.484829:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828910.484837:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.484845:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828910.484881:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672517996416/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828910.484903:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.484906:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828910.484908:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.484911:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1916 00000100:00080000:3.0:1787828910.484913:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.484917:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828910.484926:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828910.484939:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672517996544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828910.484940:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517996416:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828910.484957:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.484960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828910.484961:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.484964:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1916 00000100:00080000:3.0:1787828910.484966:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.484969:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828910.484973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828910.484982:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672517996672/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828910.484993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.484996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828910.484997:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.484999:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1916 00000100:00080000:3.0:1787828910.485002:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.485006:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828910.485009:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828910.485017:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828910.485019:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672517996800/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828910.485028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828910.485032:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828910.485033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.485038:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1916 00000100:00080000:3.0:1787828910.485042:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.485057:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828910.485063:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828910.485069:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff8b2e8803b480 x1874672517996416/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828910.485080:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672517996928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828910.485088:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517996416/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828910.485105:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828910.485109:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828910.485114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828910.485115:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787828910.485120:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828910.485121:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517996416:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828910.485131:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1916 00000100:00080000:3.0:1787828910.485136:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828910.485139:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828910.485144:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828910.485148:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517996544:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828910.485160:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672517997056/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828910.485170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828910.485173:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828910.485176:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828910.485179:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828910.485184:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1916 00000100:00080000:3.0:1787828910.485188:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828910.485193:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff8b2e880392c0 x1874672517996544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828910.485194:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828910.485202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828910.485218:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672517996544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828910.485219:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517997184/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828910.485254:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828910.485258:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828910.485262:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517996544:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828910.485281:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517996672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828910.485296:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828910.485303:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff8b2e88039e00 x1874672517996672/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828910.485311:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672517996672/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828910.485318:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828910.485320:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828910.485322:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517996672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828910.485325:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517996800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828910.485347:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828910.485353:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff8b2e8803a1c0 x1874672517996800/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828910.485362:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672517996800/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828910.485367:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828910.485369:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828910.485372:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517996800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828910.485376:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517996928:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828910.485386:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828910.485393:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff8b2e88039680 x1874672517996928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828910.485400:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672517996928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828910.485406:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828910.485408:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828910.485410:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517996928:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828910.485422:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517997056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828910.485433:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828910.485439:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff8b2e8803b480 x1874672517997056/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828910.485448:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517997056/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828910.485454:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828910.485456:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828910.485458:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517997056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828910.485462:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517997184:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828910.485472:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828910.485478:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828910/real 1787828910] req@ffff8b2e880383c0 x1874672517997184/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828910.485486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517997184/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828935 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828910.485499:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828910.485501:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828910.485503:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517997184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828910.804750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828911.316722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828911.316773:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828911.508739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828911.508762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828911.828795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828912.340777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828912.340844:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828912.532723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828912.532726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828912.852761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828913.364751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828913.364799:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828913.556738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828913.556765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828913.876843:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828914.388789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828914.388840:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828914.580767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828914.580776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828914.900770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828915.412760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787828915.412858:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828915.604767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828915.604769:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.604778:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828915.604782:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828915.604803:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1922 00000100:00100000:0.0:1787828915.604812:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828915.604813:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.604824:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828915.604866:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672517997312/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828915.604890:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.604893:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828915.604894:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828915.604897:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1922 00000100:00080000:3.0:1787828915.604899:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.604903:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828915.604914:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828915.604922:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672517997440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828915.604935:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517997312:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828915.604946:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.604950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828915.604952:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828915.604955:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1922 00000100:00080000:3.0:1787828915.604957:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.604960:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828915.604964:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828915.604974:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672517997568/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828915.604980:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.604982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828915.604983:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828915.604986:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1922 00000100:00080000:3.0:1787828915.604990:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.604992:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828915.604997:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828915.605004:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672517997696/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828915.605010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828915.605012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828915.605014:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828915.605016:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1922 00000100:00080000:3.0:1787828915.605020:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.605022:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828915.605025:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828915.605031:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672517997824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828915.605037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828915.605038:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828915.605046:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828915.605049:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828915.605055:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1922 00000100:00080000:3.0:1787828915.605060:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828915.605066:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828915.605072:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828915.605089:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672517997952/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828915.605101:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828915.605104:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff8b2e8803a580 x1874672517997312/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828915.605110:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828915.605122:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828915.605137:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1922 00000100:00080000:3.0:1787828915.605144:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828915.605146:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672517997312/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828915.605155:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828915.605162:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828915.605167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828915.605169:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828915.605176:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517997312:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828915.605192:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672517998080/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828915.605207:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517997440:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828915.605249:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828915.605260:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff8b2e8803b840 x1874672517997440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828915.605269:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672517997440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828915.605276:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828915.605278:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828915.605280:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517997440:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828915.605284:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517997568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828915.605296:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828915.605302:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff8b2e88038f00 x1874672517997568/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828915.605309:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672517997568/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828915.605315:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828915.605317:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828915.605319:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517997568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828915.605323:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517997696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828915.605333:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828915.605338:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff8b2e8803b0c0 x1874672517997696/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828915.605351:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672517997696/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828915.605356:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828915.605358:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828915.605360:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517997696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828915.605364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517997824:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828915.605382:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828915.605394:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff8b2e88039a40 x1874672517997824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828915.605407:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672517997824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828915.605417:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828915.605421:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828915.605424:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517997824:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828915.605430:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517997952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828915.605445:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828915.605453:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff8b2e880383c0 x1874672517997952/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828915.605461:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672517997952/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828915.605472:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828915.605474:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828915.605476:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517997952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828915.605489:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517998080:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828915.605501:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828915.605507:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828915/real 1787828915] req@ffff8b2e8803b480 x1874672517998080/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828915.605514:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672517998080/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828940 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828915.605520:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828915.605522:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828915.605524:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517998080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828915.924787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828916.436792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828916.436872:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828916.628771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828916.628788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828916.948760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828917.460714:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828917.460807:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828917.652733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828917.652788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828917.972746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828918.484746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828918.484787:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828918.676724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828918.676749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828918.996719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828919.508721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828919.508838:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828919.700806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828919.700811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828920.020739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828920.532752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828920.532816:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828920.724789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828920.724804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828920.724811:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828920.724835:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828920.724841:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828920.724852:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1927 00000100:00080000:2.0:1787828920.724866:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828920.724874:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828920.724912:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517998208/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828920.724941:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828920.724945:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828920.724965:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828920.724972:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1927 00000100:00080000:2.0:1787828920.724978:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828920.724983:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828920.724995:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828920.725001:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517998208:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828920.725040:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517998336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828920.725052:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828920.725056:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828920.725058:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828920.725061:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1927 00000100:00080000:2.0:1787828920.725066:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828920.725070:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828920.725076:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828920.725087:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517998464/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828920.725096:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828920.725105:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828920.725109:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828920.725112:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828920.725119:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1927 00000100:00080000:2.0:1787828920.725126:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828920.725129:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828920.725134:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828920.725147:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517998592/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828920.725157:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828920.725162:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff8b2e8801c780 x1874672517998208/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828920.725163:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828920.725169:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828920.725186:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1927 00000100:00080000:2.0:1787828920.725197:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828920.725202:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828920.725208:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828920.725210:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517998208/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828920.725223:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517998720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828920.725228:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828920.725235:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828920.725240:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828920.725242:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787828920.725244:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828920.725256:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1927 00000100:00100000:0.0:1787828920.725261:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517998208:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787828920.725263:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828920.725274:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828920.725280:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828920.725298:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517998848/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828920.725300:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517998336:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828920.725332:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828920.725338:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787828920.725355:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828920.725363:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828920.725372:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1927 00000100:00100000:0.0:1787828920.725378:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff8b2e8801ed00 x1874672517998336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828920.725381:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828920.725398:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828920.725407:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517998336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828920.725432:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787828920.725432:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828920.725438:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828920.725445:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517998336:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828920.725453:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517998464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787828920.725466:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517998976/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828920.725474:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828920.725494:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff8b2e8801cb40 x1874672517998464/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828920.725547:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517998464/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828920.725555:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828920.725558:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828920.725563:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517998464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828920.725570:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517998592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828920.725582:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828920.725587:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff8b2e8801c3c0 x1874672517998592/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828920.725594:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517998592/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828920.725600:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828920.725602:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828920.725604:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517998592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828920.725608:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517998720:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828920.725637:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828920.725673:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff8b2e8801d680 x1874672517998720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828920.725682:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517998720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828920.725689:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828920.725691:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828920.725693:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517998720:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828920.725707:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517998848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828920.725722:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828920.725729:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff8b2e8801de00 x1874672517998848/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828920.725737:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517998848/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828920.725743:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828920.725745:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828920.725747:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517998848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828920.725752:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517998976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828920.725762:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828920.725768:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828920/real 1787828920] req@ffff8b2e8801f0c0 x1874672517998976/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828920.725775:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517998976/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828945 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828920.725780:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828920.725782:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828920.725784:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517998976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828921.044737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828921.556727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828921.556807:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828921.748718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828921.748760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828922.068723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828922.580740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828922.580830:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828922.772747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828922.772747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828923.092793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828923.604709:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828923.604785:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828923.796724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828923.796724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828924.116749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828924.628745:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828924.628789:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828924.820786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828924.820786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828925.140776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828925.652787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828925.652859:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828925.844793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828925.844793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828925.844802:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828925.844813:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828925.844820:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828925.844834:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1932 00000100:00080000:2.0:1787828925.844849:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828925.844860:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828925.844913:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672517999104/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828925.844953:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828925.844958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828925.844961:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828925.844972:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1932 00000100:00080000:2.0:1787828925.844978:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828925.844984:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828925.844993:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517999104:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787828925.844999:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828925.845025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672517999232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828925.845045:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828925.845051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828925.845054:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828925.845060:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1932 00000100:00080000:2.0:1787828925.845066:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828925.845072:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828925.845079:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828925.845097:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672517999360/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828925.845112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828925.845118:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828925.845121:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828925.845129:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1932 00000100:00080000:2.0:1787828925.845135:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828925.845137:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517999232:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828925.845149:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828925.845160:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828925.845184:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672517999488/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828925.845197:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517999360:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787828925.845207:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828925.845214:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828925.845217:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828925.845225:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1932 00000100:00080000:2.0:1787828925.845232:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828925.845233:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517999488:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787828925.845243:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828925.845252:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828925.845272:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672517999616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828925.845290:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828925.845296:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828925.845299:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828925.845306:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1932 00000100:00080000:2.0:1787828925.845312:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828925.845317:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517999616:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828925.845324:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828925.845334:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828925.845356:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672517999744/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828925.845375:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828925.845381:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828925.845384:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828925.845391:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517999744:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787828925.845393:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1932 00000100:00080000:2.0:1787828925.845405:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828925.845412:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828925.845422:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828925.845444:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672517999872/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828925.845502:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517999872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828926.164719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828926.676730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828926.676809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828926.868739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828926.868834:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828927.188770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828927.700706:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828927.700778:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828927.892761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828927.892809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828928.212739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828928.724718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828928.724802:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828928.916736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828928.916927:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:0.0:1787828928.917009:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:0.0:1787828928.917076:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787828928.917089:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828928.917123:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828928.917135:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828928.917147:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787828928.917157:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787828928.917163:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828928] req@ffff8b2e8801de00 x1874672517999232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787828928.917169:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787828928.917198:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672517999232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787828928.917202:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00080000:3.0:1787828928.917225:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828928.917231:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828928.917237:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517999232:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828928.917245:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828928] req@ffff8b2e8801d680 x1874672517999360/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828928.917260:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672517999360/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828928.917269:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828928.917273:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828928.917276:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517999360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828928.917283:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828928] req@ffff8b2e8801f0c0 x1874672517999104/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828928.917297:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672517999104/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828928.917305:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828928.917309:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828928.917312:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672517999104:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828928.917336:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828928] req@ffff8b2e8801c780 x1874672517999872/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828928.917347:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672517999872/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828928.917355:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828928.917358:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828928.917362:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517999872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828928.917368:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828928] req@ffff8b2e8801ed00 x1874672517999744/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828928.917379:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672517999744/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828928.917387:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828928.917391:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828928.917394:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517999744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828928.917400:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828928] req@ffff8b2e8801cb40 x1874672517999616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828928.917413:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672517999616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828928.917422:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828928.917426:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828928.917429:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672517999616:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828928.917435:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828925/real 1787828928] req@ffff8b2e8801c3c0 x1874672517999488/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828928.917448:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672517999488/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828950 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828928.917456:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828928.917460:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828928.917463:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672517999488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828929.236741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828929.748760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828929.748842:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828929.940810:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828929.940850:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828930.260733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828930.772744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828930.772806:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787828930.964737:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828930.964747:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828930.964752:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828930.964753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828930.964771:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1937 00000100:00080000:2.0:1787828930.964786:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828930.964794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828930.964799:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828930.964841:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518000000/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828930.964868:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828930.964872:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828930.964874:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828930.964880:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1937 00000100:00080000:2.0:1787828930.964885:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828930.964890:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828930.964899:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518000000:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787828930.964903:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828930.964918:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518000128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828930.964928:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828930.964932:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828930.964934:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828930.964938:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1937 00000100:00080000:2.0:1787828930.964943:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828930.964947:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828930.964952:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828930.964965:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518000256/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828930.964975:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828930.964979:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828930.964981:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828930.964985:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1937 00000100:00080000:2.0:1787828930.964989:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828930.964993:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787828930.964998:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787828930.965000:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828930.965014:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518000384/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828930.965024:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828930.965028:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828930.965030:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828930.965035:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1937 00000100:00080000:2.0:1787828930.965039:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828930.965044:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828930.965049:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828930.965056:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff8b2e8801da40 x1874672518000000/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828930.965064:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518000512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828930.965081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828930.965085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828930.965089:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828930.965090:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518000000/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828930.965097:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1937 00000100:00080000:2.0:1787828930.965101:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828930.965106:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828930.965119:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828930.965122:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828930.965130:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828930.965135:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518000000:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828930.965139:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518000640/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828930.965151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828930.965156:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828930.965158:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828930.965162:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1937 00000100:00080000:2.0:1787828930.965167:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828930.965171:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518000128:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828930.965175:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828930.965184:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828930.965200:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518000768/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828930.965211:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828930.965235:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff8b2e8801fc00 x1874672518000128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828930.965251:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518000128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828930.965277:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828930.965281:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828930.965284:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518000128:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828930.965291:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518000256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828930.965320:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828930.965333:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff8b2e8801f480 x1874672518000256/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828930.965346:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518000256/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828930.965355:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828930.965359:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828930.965363:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518000256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828930.965369:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518000384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828930.965389:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828930.965402:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff8b2e8801cf00 x1874672518000384/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828930.965413:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518000384/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828930.965421:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828930.965424:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828930.965428:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518000384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828930.965435:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518000512:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828930.965455:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828930.965467:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff8b2e8801c000 x1874672518000512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828930.965481:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518000512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828930.965491:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828930.965494:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828930.965498:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518000512:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828930.965504:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518000640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828930.965526:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828930.965539:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff8b2e8801e580 x1874672518000640/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828930.965571:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518000640/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828930.965580:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828930.965584:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828930.965587:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518000640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828930.965719:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518000768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828930.965747:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828930.965760:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828930/real 1787828930] req@ffff8b2e8801e940 x1874672518000768/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828930.965771:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518000768/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828955 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828930.965779:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828930.965782:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828930.965786:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518000768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828931.284775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828931.796774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828931.796873:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828931.988802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828931.988853:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828932.308708:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828932.820730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828932.820823:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828933.012732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828933.012740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828933.332723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828933.844795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828933.844844:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828934.036758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828934.036766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828934.356723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828934.868722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828934.868790:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828935.060743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828935.060743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828935.380740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828935.892789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828935.892904:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828936.084777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828936.084777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787828936.084781:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828936.084791:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828936.084795:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828936.084806:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1942 00000100:00080000:2.0:1787828936.084818:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828936.084826:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828936.084869:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518000896/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828936.084911:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828936.084914:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828936.084915:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828936.084919:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1942 00000100:00080000:2.0:1787828936.084921:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828936.084925:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828936.084933:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828936.084944:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518000896:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828936.084946:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518001024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828936.084958:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828936.084960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828936.084962:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828936.084975:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1942 00000100:00080000:2.0:1787828936.084978:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828936.084980:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828936.084985:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828936.084994:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518001152/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828936.085001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828936.085005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787828936.085006:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828936.085007:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828936.085013:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1942 00000100:00080000:2.0:1787828936.085016:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828936.085019:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828936.085022:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828936.085032:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518001280/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828936.085038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828936.085041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828936.085042:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828936.085046:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1942 00000100:00080000:2.0:1787828936.085049:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828936.085054:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828936.085055:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828936/real 1787828936] req@ffff8b2e8801e940 x1874672518000896/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787828936.085058:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828936.085070:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518001408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828936.085078:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518000896/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828936.085081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828936.085089:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828936.085093:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828936.085098:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828936.085099:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1942 00000100:00080000:2.0:1787828936.085104:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828936.085107:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787828936.085111:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828936.085115:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828936.085116:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518000896:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828936.085127:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518001536/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828936.085134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828936.085136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828936.085139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828936.085142:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1942 00000100:00080000:2.0:1787828936.085147:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828936.085167:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518001024:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828936.085173:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828936.085182:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828936.085202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518001664/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828936.085202:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828936.085220:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828936/real 1787828936] req@ffff8b2e8801e580 x1874672518001024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828936.085235:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518001024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828936.085244:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828936.085247:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828936.085250:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518001024:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828936.085256:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518001152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828936.085271:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828936.085279:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828936/real 1787828936] req@ffff8b2e8801c000 x1874672518001152/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828936.085289:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518001152/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828936.085296:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828936.085298:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828936.085301:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518001152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828936.085305:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518001280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828936.085335:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828936.085343:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828936/real 1787828936] req@ffff8b2e8801cf00 x1874672518001280/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828936.085352:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518001280/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828936.085358:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828936.085360:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828936.085363:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518001280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828936.085367:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518001408:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828936.085379:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828936.085404:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828936/real 1787828936] req@ffff8b2e8801f480 x1874672518001408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828936.085419:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518001408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828936.085429:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828936.085432:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828936.085436:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518001408:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828936.085443:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518001536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828936.085464:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828936.085475:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828936/real 1787828936] req@ffff8b2e8801fc00 x1874672518001536/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828936.085498:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518001536/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828936.085532:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828936.085540:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828936.085547:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518001536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828936.085592:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518001664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828936.085613:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828936.085624:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828936/real 1787828936] req@ffff8b2e8801da40 x1874672518001664/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828936.085634:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518001664/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828961 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828936.085641:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828936.085644:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828936.085646:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518001664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828936.404707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828936.916730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828936.916778:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828937.108749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828937.108749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828937.428707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828937.940762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828937.940842:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828938.132745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828938.132749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828938.452707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828938.964749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828938.964810:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828939.156769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828939.156771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828939.476726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828939.988782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828939.988860:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828940.180768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787828940.180772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828940.500726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828941.012774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828941.012855:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828941.204761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828941.204794:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828941.204802:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828941.204808:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828941.204811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828941.204819:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1947 00000100:00080000:1.0:1787828941.204834:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828941.204847:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828941.204911:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518001792/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828941.204964:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828941.204969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828941.204972:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828941.204978:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1947 00000100:00080000:1.0:1787828941.204984:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828941.204992:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828941.205006:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828941.205021:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518001792:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828941.205033:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518001920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828941.205052:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828941.205056:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828941.205060:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828941.205066:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1947 00000100:00080000:1.0:1787828941.205072:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828941.205077:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828941.205084:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828941.205098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518002048/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828941.205114:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828941.205116:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828941.205123:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828941.205126:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828941.205134:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1947 00000100:00080000:1.0:1787828941.205141:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828941.205148:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828941.205154:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828941.205170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518002176/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828941.205185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787828941.205186:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828941/real 1787828941] req@ffff8b2e88077c00 x1874672518001792/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828941.205196:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828941.205203:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828941.205220:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1947 00000100:00080000:1.0:1787828941.205229:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828941.205235:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828941.205240:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518001792/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828941.205244:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787828941.205267:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828941.205274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518002304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828941.205280:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828941.205295:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518001792:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828941.205312:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828941.205318:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828941.205321:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828941.205328:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1947 00000100:00080000:1.0:1787828941.205332:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828941.205337:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828941.205344:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828941.205351:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518001920:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828941.205366:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518002432/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828941.205413:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828941.205419:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828941.205423:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828941.205434:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1947 00000800:00000100:3.0:1787828941.205434:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828941.205448:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828941.205456:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828941.205467:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828941.205467:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828941/real 1787828941] req@ffff8b2e88077480 x1874672518001920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828941.205536:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518001920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828941.205538:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518002560/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828941.205588:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828941.205597:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828941.205604:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518001920:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828941.205616:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518002048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828941.205650:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828941.205700:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828941/real 1787828941] req@ffff8b2e88075a40 x1874672518002048/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828941.205718:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518002048/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828941.205729:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828941.205732:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828941.205737:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518002048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828941.205745:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518002176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828941.205768:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828941.205781:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828941/real 1787828941] req@ffff8b2e88075680 x1874672518002176/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828941.205796:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518002176/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828941.205806:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828941.205810:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828941.205831:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518002176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828941.205837:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518002304:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828941.205854:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828941.205864:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828941/real 1787828941] req@ffff8b2e88074f00 x1874672518002304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828941.205875:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518002304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828941.205884:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828941.205887:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828941.205890:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518002304:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828941.205909:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518002432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828941.205925:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828941.205934:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828941/real 1787828941] req@ffff8b2e88076940 x1874672518002432/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828941.205946:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518002432/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828941.205954:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828941.205957:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828941.205960:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518002432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828941.205966:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518002560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828941.205980:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828941.205989:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828941/real 1787828941] req@ffff8b2e88074780 x1874672518002560/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828941.206000:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518002560/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828966 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828941.206008:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828941.206010:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828941.206014:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518002560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828941.524749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828942.036778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828942.036871:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828942.228741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828942.228801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828942.548767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828943.060729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828943.060851:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828943.252736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828943.252737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828943.572743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828944.084766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828944.084809:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828944.276760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828944.276760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828944.596750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828945.108729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828945.108823:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828945.300773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828945.300789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828945.620735:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828946.132820:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828946.133021:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828946.324728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828946.324729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787828946.324752:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828946.324758:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787828946.324763:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828946.324771:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1952 00000100:00080000:0.0:1787828946.324781:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828946.324787:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828946.324822:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518002688/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828946.324844:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828946.324847:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828946.324849:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828946.324852:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1952 00000100:00080000:0.0:1787828946.324855:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828946.324857:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828946.324866:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828946.324875:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518002688:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828946.324881:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518002816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828946.324890:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828946.324893:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828946.324895:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828946.324898:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1952 00000100:00080000:0.0:1787828946.324900:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828946.324903:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828946.324906:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787828946.324916:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518002944/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828946.324923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828946.324925:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828946.324926:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828946.324929:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1952 00000100:00080000:0.0:1787828946.324931:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787828946.324934:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828946.324938:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828946.324942:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828946.324951:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518003072/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828946.324961:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828946.324966:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787828946.324968:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828946.324974:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1952 00000100:00080000:0.0:1787828946.324979:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787828946.324980:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff8b2e880403c0 x1874672518002688/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828946.324989:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828946.324999:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828946.325002:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518002688/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828946.325012:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828946.325019:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828946.325021:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518003200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787828946.325022:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518002688:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787828946.325038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828946.325041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787828946.325043:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828946.325047:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1952 00000100:00100000:3.0:1787828946.325051:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518002816:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787828946.325052:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828946.325061:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828946.325067:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828946.325078:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518003328/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787828946.325085:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787828946.325094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787828946.325097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828946.325099:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787828946.325103:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff8b2e88042940 x1874672518002816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828946.325104:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1952 00000100:00080000:0.0:1787828946.325109:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828946.325116:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787828946.325123:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828946.325128:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518002816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828946.325135:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518003456/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828946.325140:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828946.325148:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828946.325153:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518002816:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828946.325158:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518002944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828946.325173:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828946.325181:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff8b2e88043480 x1874672518002944/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828946.325189:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518002944/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828946.325195:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828946.325197:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828946.325199:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518002944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828946.325204:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518003072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828946.325214:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828946.325220:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff8b2e880430c0 x1874672518003072/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828946.325243:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518003072/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828946.325249:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828946.325251:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828946.325254:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518003072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828946.325257:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518003200:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787828946.325269:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828946.325277:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff8b2e88040000 x1874672518003200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828946.325285:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518003200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828946.325291:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828946.325292:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828946.325294:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518003200:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828946.325306:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518003328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828946.325317:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828946.325324:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff8b2e88041a40 x1874672518003328/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828946.325332:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518003328/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828946.325342:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828946.325344:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828946.325346:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518003328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828946.325350:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518003456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787828946.325360:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828946.325365:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828946/real 1787828946] req@ffff8b2e88040f00 x1874672518003456/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828946.325373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518003456/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828971 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828946.325379:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828946.325380:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787828946.325382:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518003456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828946.644776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828947.156770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828947.157012:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828947.348778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828947.348778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828947.668704:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828948.180819:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828948.180898:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828948.372727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828948.372754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828948.692746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828949.204718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828949.204873:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828949.396825:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828949.396827:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828949.716755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828950.228769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828950.228827:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828950.420752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828950.420759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828950.740753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828951.252783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828951.252864:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828951.444750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828951.444755:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828951.444765:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828951.444769:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828951.444772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828951.444781:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1957 00000100:00080000:3.0:1787828951.444794:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828951.444802:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828951.444842:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518003584/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828951.444872:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828951.444876:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828951.444879:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828951.444884:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1957 00000100:00080000:3.0:1787828951.444889:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828951.444894:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828951.444901:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518003584:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787828951.444905:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828951.444924:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518003712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828951.444937:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828951.444941:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828951.444944:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828951.444948:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1957 00000100:00080000:3.0:1787828951.444953:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787828951.444954:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828951.444962:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828951.444970:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828951.444985:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518003840/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828951.444997:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff8b2e88039680 x1874672518003584/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828951.444997:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828951.445007:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828951.445011:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828951.445018:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1957 00000100:00100000:0.0:1787828951.445021:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518003584/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828951.445024:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828951.445033:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828951.445035:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787828951.445041:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828951.445043:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828951.445050:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518003584:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828951.445058:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518003968/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828951.445070:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828951.445073:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787828951.445077:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518003712:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787828951.445077:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828951.445088:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1957 00000100:00080000:3.0:1787828951.445093:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828951.445098:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828951.445103:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828951.445111:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828951.445129:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff8b2e8803a1c0 x1874672518003712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828951.445130:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518004096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828951.445160:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518003712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828951.445162:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828951.445171:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828951.445176:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828951.445180:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828951.445184:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1957 00000100:00080000:0.0:1787828951.445189:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787828951.445192:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828951.445197:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518003712:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787828951.445202:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828951.445209:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518003840:192.168.203.101@tcp:8:kworker.0 00000008:00100000:3.0:1787828951.445210:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828951.445231:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518004224/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828951.445242:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828951.445246:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828951.445251:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828951.445254:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828951.445258:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff8b2e88039e00 x1874672518003840/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828951.445261:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1957 00000100:00080000:3.0:1787828951.445269:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828951.445277:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828951.445283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518003840/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828951.445286:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828951.445306:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518004352/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828951.445307:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.445317:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828951.445326:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518003840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828951.445334:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518003968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828951.445355:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828951.445365:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff8b2e880392c0 x1874672518003968/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828951.445376:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518003968/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828951.445384:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.445387:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828951.445390:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518003968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828951.445419:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518004096:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828951.445437:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828951.445446:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff8b2e88039680 x1874672518004096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828951.445457:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518004096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828951.445466:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.445468:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828951.445472:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518004096:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828951.445477:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518004224:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828951.445490:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828951.445498:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff8b2e8803b480 x1874672518004224/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828951.445508:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518004224/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828951.445515:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.445518:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828951.445521:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518004224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828951.445526:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518004352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828951.445538:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828951.445546:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828951/real 1787828951] req@ffff8b2e880383c0 x1874672518004352/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828951.445556:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518004352/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828976 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828951.445563:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828951.445565:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828951.445568:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518004352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828951.764765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828952.276728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828952.276799:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828952.468754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828952.468759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828952.788755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828953.300796:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828953.300860:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828953.492714:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828953.492726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828953.812737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828954.324741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828954.324889:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828954.516742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828954.516742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828954.836784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828955.348769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828955.348822:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828955.540762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828955.540794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828955.860722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787828956.372802:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787828956.372872:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787828956.564736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828956.564737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828956.564763:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828956.564771:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828956.564776:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828956.564786:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1962 00000100:00080000:1.0:1787828956.564798:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828956.564806:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828956.564846:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518004480/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828956.564873:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828956.564877:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828956.564879:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828956.564884:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1962 00000100:00080000:1.0:1787828956.564896:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828956.564901:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518004480:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828956.564905:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828956.564917:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828956.564933:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518004608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828956.564946:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828956.564950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828956.564952:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787828956.564957:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828956.564958:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1962 00000100:00080000:1.0:1787828956.564966:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828956.564971:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828956.564977:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828956.564990:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518004736/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828956.565001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828956.565007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787828956.565008:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff8b2e88077840 x1874672518004480/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828956.565010:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828956.565018:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1962 00000100:00080000:1.0:1787828956.565022:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828956.565025:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828956.565032:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828956.565046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518004864/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828956.565054:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518004480/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828956.565061:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828956.565070:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828956.565075:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828956.565079:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787828956.565081:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1962 00000100:00080000:0.0:1787828956.565087:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787828956.565088:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828956.565095:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518004480:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828956.565097:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828956.565107:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828956.565120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518004992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828956.565128:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518004608:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828956.565134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828956.565139:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828956.565142:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828956.565147:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1962 00000800:00000100:0.0:1787828956.565159:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828956.565160:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828956.565167:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828956.565173:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff8b2e88076580 x1874672518004608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787828956.565181:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828956.565186:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518004608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828956.565195:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.565200:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787828956.565200:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518005120/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828956.565206:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518004608:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828956.565216:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518004736:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787828956.565224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828956.565229:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828956.565232:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828956.565238:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1962 00000800:00000100:0.0:1787828956.565240:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828956.565246:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828956.565252:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff8b2e88076d00 x1874672518004736/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828956.565254:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828956.565264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828956.565268:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518004736/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828956.565276:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.565279:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828956.565284:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518004736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828956.565284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518005248/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828956.565295:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518004864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828956.565314:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828956.565321:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff8b2e880770c0 x1874672518004864/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828956.565330:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518004864/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828956.565335:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.565337:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828956.565339:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518004864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828956.565349:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518004992:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828956.565360:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828956.565379:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff8b2e88074000 x1874672518004992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828956.565392:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518004992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828956.565398:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.565400:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828956.565402:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518004992:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828956.565406:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518005120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828956.565417:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828956.565424:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff8b2e880761c0 x1874672518005120/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828956.565431:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518005120/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828956.565436:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.565438:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828956.565440:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518005120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828956.565443:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518005248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828956.565452:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828956.565458:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828956/real 1787828956] req@ffff8b2e88075e00 x1874672518005248/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828956.565465:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518005248/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828981 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828956.565474:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828956.565476:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828956.565478:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518005248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828956.884739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828957.396753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828957.396833:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828957.588754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828957.588754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828957.908743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828958.420746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828958.420796:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828958.612751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828958.612771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828958.932757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828959.444787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828959.444863:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828959.636789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828959.636800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828959.956750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828960.468718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828960.468775:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828960.660807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828960.660815:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828960.980737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828961.492738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828961.492873:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787828961.684723:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828961.684731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828961.684731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787828961.684733:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828961.684740:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828961.684750:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1968 00000100:00080000:1.0:1787828961.684759:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828961.684766:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828961.684806:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518005376/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828961.684827:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828961.684830:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828961.684831:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828961.684834:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1968 00000100:00080000:1.0:1787828961.684837:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828961.684840:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828961.684848:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828961.684859:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518005376:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828961.684861:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518005504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828961.684872:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828961.684875:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828961.684876:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828961.684879:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1968 00000100:00080000:1.0:1787828961.684881:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828961.684883:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828961.684886:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828961.684894:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518005632/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828961.684900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828961.684902:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828961.684904:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828961.684907:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1968 00000100:00080000:1.0:1787828961.684910:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828961.684913:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787828961.684917:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787828961.684919:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828961.684929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518005760/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828961.684937:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828961.684939:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828961.684940:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828961.684943:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1968 00000100:00080000:1.0:1787828961.684945:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828961.684947:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828961.684950:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828961.684959:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518005888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828961.684962:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff8b2e88074b40 x1874672518005376/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828961.684978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828961.684982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828961.684983:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828961.684987:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1968 00000100:00100000:0.0:1787828961.684990:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518005376/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828961.684991:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828961.684996:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828961.685002:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828961.685008:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787828961.685012:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518006016/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828961.685016:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828961.685023:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518005376:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787828961.685027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828961.685031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828961.685033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828961.685036:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1968 00000100:00080000:1.0:1787828961.685038:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828961.685041:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828961.685047:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518005504:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787828961.685049:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828961.685062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518006144/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828961.685071:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828961.685084:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff8b2e88075e00 x1874672518005504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.685093:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518005504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.685099:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.685102:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828961.685104:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518005504:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828961.685108:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518005632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828961.685119:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828961.685125:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff8b2e880761c0 x1874672518005632/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.685133:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518005632/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.685138:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.685140:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828961.685142:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518005632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828961.685146:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518005760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828961.685155:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828961.685161:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff8b2e88074000 x1874672518005760/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.685168:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518005760/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.685173:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.685175:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828961.685177:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518005760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828961.685181:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518005888:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828961.685191:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828961.685197:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff8b2e880770c0 x1874672518005888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.685204:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518005888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.685210:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.685212:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828961.685213:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518005888:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828961.685217:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518006016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828961.685226:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828961.685232:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff8b2e88076d00 x1874672518006016/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.685239:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518006016/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.685244:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.685246:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828961.685248:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518006016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828961.685259:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518006144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828961.685269:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828961.685275:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828961/real 1787828961] req@ffff8b2e88076580 x1874672518006144/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828961.685283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518006144/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828986 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828961.685288:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828961.685290:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828961.685291:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518006144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828962.004724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828962.516730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828962.516758:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828962.708721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828962.708727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828963.028734:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828963.540741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828963.540815:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828963.732780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828963.732834:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828964.052724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828964.564712:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828964.564765:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828964.756759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828964.756766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828965.076763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828965.588762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828965.588830:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828965.780735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828965.780756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828966.100756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828966.612727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828966.612809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828966.804760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828966.804766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787828966.804786:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828966.804791:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787828966.804795:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828966.804805:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1973 00000100:00080000:1.0:1787828966.804814:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828966.804820:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787828966.804855:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518006272/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828966.804874:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828966.804876:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828966.804878:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828966.804881:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1973 00000100:00080000:1.0:1787828966.804884:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828966.804887:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828966.804895:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828966.804908:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518006272:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828966.804910:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518006400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828966.804924:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828966.804928:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828966.804930:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828966.804934:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1973 00000100:00080000:1.0:1787828966.804938:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828966.804942:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828966.804949:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787828966.804962:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518006528/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828966.804971:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787828966.804976:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828966.804977:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828966.804981:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828966.804987:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1973 00000100:00080000:1.0:1787828966.804991:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828966.804995:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828966.805000:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828966.805014:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518006656/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828966.805018:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828966] req@ffff8b2e88077840 x1874672518006272/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828966.805027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828966.805033:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787828966.805035:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828966.805040:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1973 00000100:00100000:0.0:1787828966.805042:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518006272/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828966.805044:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828966.805052:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828966.805057:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787828966.805058:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828966.805064:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828966.805068:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518006272:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787828966.805072:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518006784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787828966.805080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828966.805082:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787828966.805083:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828966.805086:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1973 00000100:00080000:1.0:1787828966.805088:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828966.805091:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518006400:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787828966.805094:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828966.805100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787828966.805110:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518006912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828966.805118:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787828966.805118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787828966.805124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787828966.805126:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787828966.805130:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1973 00000100:00100000:0.0:1787828966.805135:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828966] req@ffff8b2e88074780 x1874672518006400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787828966.805135:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787828966.805142:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787828966.805149:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828966.805155:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518006400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787828966.805160:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518007040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828966.805165:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828966.805173:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828966.805178:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518006400:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828966.805182:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518006528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828966.805197:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828966.805204:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828966] req@ffff8b2e88076940 x1874672518006528/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828966.805211:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518006528/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828966.805217:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828966.805219:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828966.805221:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518006528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828966.805224:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518006656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828966.805234:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828966.805239:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828966] req@ffff8b2e88074f00 x1874672518006656/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828966.805246:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518006656/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828966.805252:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828966.805254:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828966.805256:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518006656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828966.805259:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518006784:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828966.805269:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828966.805275:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828966] req@ffff8b2e88075680 x1874672518006784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828966.805289:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518006784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828966.805308:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828966.805312:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828966.805315:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518006784:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828966.805332:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518006912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828966.805351:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828966.805359:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828966] req@ffff8b2e88075a40 x1874672518006912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828966.805367:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518006912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828966.805372:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828966.805374:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828966.805377:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518006912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828966.805381:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518007040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828966.805391:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828966.805397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828966/real 1787828966] req@ffff8b2e88077480 x1874672518007040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828966.805404:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518007040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828991 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828966.805409:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828966.805410:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828966.805412:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518007040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828967.124812:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828967.636778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828967.636837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828967.828728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828967.828748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828968.148757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828968.660710:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828968.660807:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828968.852765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828968.852766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828969.172691:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828969.684736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828969.684792:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828969.876716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828969.876716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828970.196733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828970.708734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828970.708784:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828970.900766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787828970.900823:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828971.220723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828971.732780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828971.732895:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828971.924782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828971.924785:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828971.924796:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828971.924801:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828971.924812:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1978 00000100:00080000:3.0:1787828971.924824:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828971.924832:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828971.924872:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787828971.924879:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518007168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828971.924908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828971.924912:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828971.924915:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828971.924920:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1978 00000100:00080000:3.0:1787828971.924924:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828971.924928:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828971.924940:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828971.924963:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518007168:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828971.924964:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518007296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828971.924990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828971.924996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828971.924999:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828971.925005:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1978 00000100:00080000:3.0:1787828971.925013:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828971.925020:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828971.925029:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828971.925048:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518007424/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828971.925061:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828971.925067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828971.925074:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828971.925078:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828971.925086:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1978 00000100:00080000:3.0:1787828971.925093:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828971.925100:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828971.925109:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828971.925120:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff8b2e88039a40 x1874672518007168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828971.925134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518007552/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828971.925147:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518007168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828971.925149:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828971.925160:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828971.925166:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787828971.925179:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828971.925179:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1978 00000100:00080000:3.0:1787828971.925189:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828971.925191:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787828971.925202:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828971.925203:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518007168:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787828971.925214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828971.925228:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518007680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828971.925237:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828971.925241:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828971.925243:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828971.925251:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518007296:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787828971.925252:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1978 00000100:00080000:3.0:1787828971.925284:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828971.925291:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828971.925298:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828971.925312:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828971.925317:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518007808/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828971.925332:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828971.925337:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff8b2e8803b0c0 x1874672518007296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828971.925338:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828971.925343:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828971.925352:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1978 00000100:00080000:3.0:1787828971.925359:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828971.925366:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828971.925374:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828971.925385:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518007296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787828971.925387:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518007936/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828971.925435:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.925452:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828971.925457:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518007296:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828971.925468:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518007424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828971.925520:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828971.925543:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff8b2e88038f00 x1874672518007424/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828971.925562:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518007424/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828971.925571:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.925574:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828971.925578:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518007424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828971.925589:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518007552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828971.925630:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828971.925688:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff8b2e8803b840 x1874672518007552/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828971.925708:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518007552/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828971.925719:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.925724:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828971.925729:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518007552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828971.925737:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518007680:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828971.925767:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828971.925785:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff8b2e8803a580 x1874672518007680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828971.925802:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518007680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828971.925814:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.925819:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828971.925825:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518007680:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828971.925852:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518007808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828971.925874:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828971.925886:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff8b2e88039a40 x1874672518007808/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828971.925903:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518007808/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828971.925916:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.925920:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828971.925926:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518007808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828971.925936:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518007936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828971.925966:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828971.925984:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828971/real 1787828971] req@ffff8b2e880383c0 x1874672518007936/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828971.926001:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518007936/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787828996 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828971.926013:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828971.926019:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828971.926024:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518007936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828972.244722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828972.756788:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828972.756850:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828972.948749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828972.948749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828973.268709:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828973.780730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828973.780795:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828973.972740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828973.972740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828974.292728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828974.804749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828974.804807:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828974.996762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828974.996763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828975.316705:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828975.828715:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828975.828797:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828976.020760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828976.020770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828976.340736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828976.852743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828976.852787:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828977.044708:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828977.044711:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828977.044719:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828977.044722:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828977.044727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828977.044740:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1983 00000100:00080000:3.0:1787828977.044751:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828977.044757:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828977.044788:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518008064/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828977.044806:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828977.044809:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828977.044810:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828977.044814:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1983 00000100:00080000:3.0:1787828977.044816:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828977.044818:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828977.044827:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828977.044836:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518008064:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828977.044838:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518008192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828977.044849:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828977.044851:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828977.044852:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828977.044854:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1983 00000100:00080000:3.0:1787828977.044857:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828977.044859:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828977.044862:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828977.044870:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518008320/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828977.044876:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828977.044878:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828977.044879:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828977.044882:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1983 00000100:00080000:3.0:1787828977.044885:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828977.044887:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828977.044890:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828977.044898:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828977.044899:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518008448/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828977.044908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828977.044911:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828977.044913:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828977.044916:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1983 00000100:00080000:3.0:1787828977.044918:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828977.044920:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828977.044922:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828977.044929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518008576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828977.044935:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828977.044939:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828977.044941:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828977/real 1787828977] req@ffff8b2e8803b480 x1874672518008064/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828977.044942:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828977.044949:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1983 00000100:00080000:3.0:1787828977.044952:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828977.044958:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828977.044963:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828977.044973:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518008704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828977.044977:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518008064/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828977.044986:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828977.044996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787828977.045000:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828977.045000:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828977.045010:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1983 00000100:00080000:0.0:1787828977.045011:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828977.045017:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518008064:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828977.045019:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828977.045033:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828977.045038:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518008192:192.168.203.101@tcp:38:kworker.0 00000008:00100000:3.0:1787828977.045038:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828977.045056:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518008832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828977.045067:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828977.045079:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828977/real 1787828977] req@ffff8b2e88039680 x1874672518008192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828977.045088:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518008192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828977.045095:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828977.045097:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828977.045099:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518008192:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828977.045103:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518008320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828977.045113:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828977.045119:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828977/real 1787828977] req@ffff8b2e880392c0 x1874672518008320/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828977.045127:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518008320/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828977.045132:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828977.045134:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828977.045135:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518008320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828977.045139:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518008448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828977.045148:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828977.045154:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828977/real 1787828977] req@ffff8b2e88039e00 x1874672518008448/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828977.045161:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518008448/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828977.045167:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828977.045169:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828977.045170:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518008448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828977.045174:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518008576:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828977.045184:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828977.045189:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828977/real 1787828977] req@ffff8b2e8803a1c0 x1874672518008576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828977.045196:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518008576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828977.045201:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828977.045203:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828977.045205:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518008576:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828977.045208:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518008704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828977.045217:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828977.045223:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828977/real 1787828977] req@ffff8b2e880383c0 x1874672518008704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828977.045246:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518008704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828977.045252:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828977.045254:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828977.045256:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518008704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828977.045269:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518008832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828977.045283:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828977.045290:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828977/real 1787828977] req@ffff8b2e88039a40 x1874672518008832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828977.045297:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518008832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829002 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828977.045303:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828977.045305:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828977.045306:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518008832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828977.364729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828977.876763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828977.876824:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828978.068748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828978.068748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828978.388727:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828978.900788:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828978.900891:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828979.092735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828979.092761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828979.412699:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828979.924762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828979.924795:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828980.116724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828980.116724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828980.436746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828980.948755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828980.948801:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828981.140727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828981.140758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828981.460760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828981.972749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828981.972822:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828982.164725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828982.164730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828982.164730:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828982.164737:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828982.164741:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828982.164751:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1988 00000100:00080000:3.0:1787828982.164763:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828982.164771:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828982.164809:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518008960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828982.164831:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828982.164834:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828982.164836:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828982.164839:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1988 00000100:00080000:3.0:1787828982.164842:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828982.164846:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828982.164855:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828982.164864:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518009088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828982.164872:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828982.164874:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518008960:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787828982.164875:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828982.164878:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828982.164885:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1988 00000100:00080000:3.0:1787828982.164887:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828982.164889:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828982.164892:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828982.164903:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518009216/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828982.164909:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828982.164911:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828982.164913:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828982.164916:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1988 00000100:00080000:3.0:1787828982.164918:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828982.164920:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828982.164923:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828982.164931:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518009344/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828982.164936:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828982.164944:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828982.164947:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828982.164948:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828982.164952:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1988 00000100:00080000:3.0:1787828982.164955:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828982.164957:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828982.164960:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828982.164967:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518009472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828982.164974:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828982.164976:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828982.164978:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828982.164980:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1988 00000100:00080000:3.0:1787828982.164983:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828982.164987:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787828982.164988:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828982/real 1787828982] req@ffff8b2e8803a580 x1874672518008960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787828982.164990:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828982.165002:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518009600/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828982.165008:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828982.165010:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518008960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828982.165011:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828982.165015:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828982.165040:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1988 00000100:00080000:0.0:1787828982.165045:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787828982.165045:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828982.165053:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828982.165055:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:3.0:1787828982.165059:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828982.165062:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518008960:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828982.165071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518009728/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828982.165093:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518009088:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828982.165117:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828982.165128:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828982/real 1787828982] req@ffff8b2e8803b840 x1874672518009088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828982.165143:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518009088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828982.165154:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828982.165158:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828982.165162:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518009088:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828982.165169:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518009216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828982.165187:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828982.165211:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828982/real 1787828982] req@ffff8b2e88038f00 x1874672518009216/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828982.165220:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518009216/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828982.165233:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828982.165237:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828982.165242:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518009216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828982.165249:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518009344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828982.165271:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828982.165281:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828982/real 1787828982] req@ffff8b2e8803b0c0 x1874672518009344/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828982.165290:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518009344/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828982.165296:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828982.165298:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828982.165300:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518009344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828982.165305:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518009472:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828982.165320:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828982.165332:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828982/real 1787828982] req@ffff8b2e88039a40 x1874672518009472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828982.165347:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518009472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828982.165357:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828982.165361:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828982.165365:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518009472:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828982.165371:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518009600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828982.165387:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828982.165396:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828982/real 1787828982] req@ffff8b2e880383c0 x1874672518009600/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828982.165404:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518009600/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828982.165412:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828982.165415:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828982.165419:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518009600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828982.165426:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518009728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828982.165461:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828982.165472:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828982/real 1787828982] req@ffff8b2e8803a1c0 x1874672518009728/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828982.165486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518009728/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829007 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828982.165495:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828982.165498:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828982.165502:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518009728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828982.484719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828982.996717:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828982.996789:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828983.188734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828983.188734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828983.508746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828984.020739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828984.020801:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828984.212779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828984.212797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828984.532725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828985.044751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828985.044827:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828985.236731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828985.236747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828985.556791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828986.068727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828986.068787:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828986.260734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828986.260734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828986.580716:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828987.092731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828987.092786:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828987.284731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828987.284731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787828987.284737:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828987.284745:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828987.284750:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828987.284760:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1993 00000100:00080000:3.0:1787828987.284770:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828987.284775:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828987.284819:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518009856/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828987.284847:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828987.284850:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828987.284851:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828987.284855:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1993 00000100:00080000:3.0:1787828987.284858:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828987.284862:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828987.284872:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828987.284875:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518009856:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828987.284885:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518009984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828987.284900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828987.284902:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828987.284904:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828987.284907:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1993 00000100:00080000:3.0:1787828987.284909:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828987.284911:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828987.284915:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828987.284924:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518010112/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828987.284929:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787828987.284937:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828987.284940:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828987.284941:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828987.284945:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1993 00000100:00080000:3.0:1787828987.284947:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828987.284949:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828987.284952:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828987.284960:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518010240/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828987.284967:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828987.284970:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff8b2e88039e00 x1874672518009856/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828987.284971:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828987.284974:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828987.284987:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1993 00000100:00100000:0.0:1787828987.284990:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518009856/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828987.284995:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828987.285003:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787828987.285006:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.285011:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:3.0:1787828987.285011:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828987.285017:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518009856:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828987.285026:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518010368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828987.285034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828987.285036:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787828987.285039:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518009984:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787828987.285039:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828987.285045:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1993 00000100:00080000:3.0:1787828987.285048:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828987.285051:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828987.285054:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787828987.285063:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787828987.285064:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518010496/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828987.285074:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828987.285076:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff8b2e880392c0 x1874672518009984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828987.285078:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828987.285081:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828987.285088:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1993 00000100:00080000:3.0:1787828987.285093:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828987.285098:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518009984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787828987.285098:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828987.285104:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828987.285112:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787828987.285116:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518010624/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828987.285117:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828987.285124:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518009984:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828987.285136:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518010112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828987.285166:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828987.285175:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff8b2e88039680 x1874672518010112/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.285183:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518010112/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.285189:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.285191:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828987.285193:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518010112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828987.285197:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518010240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828987.285209:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828987.285215:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff8b2e8803b480 x1874672518010240/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.285222:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518010240/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.285227:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.285229:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828987.285231:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518010240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828987.285234:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518010368:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828987.285244:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828987.285249:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff8b2e8803a1c0 x1874672518010368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.285261:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518010368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.285266:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.285268:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828987.285270:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518010368:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828987.285281:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518010496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828987.285291:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828987.285296:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff8b2e880383c0 x1874672518010496/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.285304:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518010496/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.285309:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.285311:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828987.285312:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518010496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828987.285316:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518010624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828987.285325:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828987.285332:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828987/real 1787828987] req@ffff8b2e88039a40 x1874672518010624/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828987.285343:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518010624/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829012 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828987.285358:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828987.285361:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828987.285364:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518010624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828987.604756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828988.116738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828988.116815:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828988.308751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828988.308755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828988.628740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828989.140753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828989.140878:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828989.332776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828989.332787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828989.652793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828990.164716:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828990.164815:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828990.356766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828990.356791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828990.676780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787828991.188770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828991.188864:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828991.380735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828991.380770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828991.700722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787828992.212868:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787828992.212920:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:3.0:1787828992.404819:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828992.404834:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787828992.404843:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787828992.404844:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828992.404847:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787828992.404858:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 1998 00000100:00080000:3.0:1787828992.404872:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828992.404881:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787828992.404926:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518010752/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828992.404957:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828992.404961:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828992.404963:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828992.404968:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1998 00000100:00080000:3.0:1787828992.404973:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828992.404978:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828992.404989:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828992.404996:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518010752:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787828992.405006:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518010880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828992.405028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828992.405032:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828992.405034:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828992.405039:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1998 00000100:00080000:3.0:1787828992.405043:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828992.405047:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828992.405053:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787828992.405068:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518011008/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828992.405079:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828992.405083:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828992.405085:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828992.405089:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518010880:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787828992.405093:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 1998 00000100:00080000:3.0:1787828992.405100:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828992.405105:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828992.405111:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828992.405137:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518011008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787828992.405143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518011136/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828992.405156:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828992.405161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787828992.405163:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828992.405169:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518011136:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787828992.405170:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1998 00000100:00080000:3.0:1787828992.405179:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828992.405185:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828992.405192:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828992.405206:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518011264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828992.405224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787828992.405228:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787828992.405230:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828992.405234:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1998 00000100:00080000:3.0:1787828992.405239:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828992.405243:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828992.405249:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828992.405250:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518011264:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787828992.405267:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518011392/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787828992.405281:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787828992.405287:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518011392:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787828992.405288:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787828992.405294:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787828992.405301:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 1998 00000100:00080000:3.0:1787828992.405305:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787828992.405310:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787828992.405316:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787828992.405328:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518011520/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828992.405374:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518011520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828992.724747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828993.236731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828993.236835:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787828993.428730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787828993.428788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787828993.748776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828994.260752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828994.260825:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828994.452757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828994.452760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828994.772715:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828995.284716:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828995.284760:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828995.476732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828995.476735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:2.0:1787828995.476944:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787828995.477002:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787828995.477025:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828995.477085:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828995.477113:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828995.477126:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828995.477139:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787828995.477152:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787828995.477161:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828995] req@ffff8b2e8803a580 x1874672518011136/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787828995.477163:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787828995.477202:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518011136/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828995.477225:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828995.477231:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828995.477236:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518011136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828995.477245:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828995] req@ffff8b2e88038f00 x1874672518010880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828995.477257:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518010880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828995.477267:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828995.477271:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828995.477274:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518010880:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828995.477282:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828995] req@ffff8b2e8803b840 x1874672518011008/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828995.477294:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518011008/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828995.477303:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828995.477307:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828995.477310:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518011008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828995.477317:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828995] req@ffff8b2e8803b0c0 x1874672518010752/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828995.477329:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518010752/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828995.477337:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828995.477341:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828995.477344:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518010752:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787828995.477374:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828995] req@ffff8b2e8803a1c0 x1874672518011520/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828995.477388:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518011520/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828995.477395:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828995.477399:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828995.477403:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518011520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828995.477410:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828995] req@ffff8b2e880383c0 x1874672518011392/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828995.477422:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518011392/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828995.477430:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828995.477433:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828995.477436:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518011392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828995.477443:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828992/real 1787828995] req@ffff8b2e88039a40 x1874672518011264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828995.477453:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518011264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829017 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828995.477461:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828995.477464:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828995.477468:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518011264:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787828995.796764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828996.308764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828996.308826:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828996.500738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828996.500738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828996.820725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828997.332725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787828997.332760:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828997.524735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828997.524735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787828997.524855:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.524861:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787828997.524865:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.524874:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2003 00000100:00080000:2.0:1787828997.524884:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.524890:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787828997.524919:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518011648/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828997.524953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.524957:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828997.524959:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.524965:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2003 00000100:00080000:2.0:1787828997.524969:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.524974:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828997.524985:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828997.524996:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518011648:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828997.525002:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518011776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828997.525022:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.525026:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828997.525028:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.525032:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2003 00000100:00080000:2.0:1787828997.525037:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.525042:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828997.525047:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787828997.525061:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518011904/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828997.525067:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828997.525094:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.525098:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828997.525100:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.525105:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2003 00000100:00080000:2.0:1787828997.525110:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.525115:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828997.525121:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828997.525137:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518012032/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787828997.525140:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff8b2e8801c3c0 x1874672518011648/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828997.525162:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.525167:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787828997.525170:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828997.525174:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518011648/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828997.525179:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2003 00000100:00080000:2.0:1787828997.525187:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787828997.525195:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787828997.525195:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828997.525202:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787828997.525204:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828997.525211:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518011648:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787828997.525221:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518012160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787828997.525232:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.525236:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787828997.525238:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787828997.525242:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518011776:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787828997.525246:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2003 00000100:00080000:2.0:1787828997.525253:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787828997.525257:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828997.525263:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787828997.525277:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518012288/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787828997.525281:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787828997.525296:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787828997.525299:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787828997.525301:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787828997.525305:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2003 00000100:00080000:2.0:1787828997.525310:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787828997.525311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff8b2e8801cb40 x1874672518011776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787828997.525318:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787828997.525330:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787828997.525336:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518011776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787828997.525347:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518012416/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787828997.525350:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.525360:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828997.525367:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518011776:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828997.525381:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518011904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828997.525407:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828997.525421:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff8b2e8801ed00 x1874672518011904/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.525434:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518011904/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.525460:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.525464:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828997.525467:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518011904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828997.525474:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518012032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828997.525497:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828997.525510:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff8b2e8801c780 x1874672518012032/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.525522:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518012032/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.525529:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.525533:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828997.525535:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518012032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828997.525542:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518012160:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787828997.525571:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828997.525584:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff8b2e8801f0c0 x1874672518012160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.525597:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518012160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.525608:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.525611:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828997.525614:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518012160:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787828997.525636:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518012288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828997.526573:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828997.526593:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff8b2e8801d680 x1874672518012288/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.526607:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518012288/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.526616:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.526620:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828997.526623:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518012288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787828997.526631:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518012416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787828997.526676:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787828997.526689:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787828997/real 1787828997] req@ffff8b2e8801de00 x1874672518012416/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787828997.526709:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518012416/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829022 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787828997.526718:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787828997.526722:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787828997.526725:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518012416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787828997.844729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828998.356748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787828998.356812:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828998.548747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828998.548747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828998.868725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787828999.380742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787828999.380846:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787828999.572729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787828999.572729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787828999.892706:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829000.404767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829000.404802:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829000.596772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829000.596772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829000.916694:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829001.428760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829001.428903:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829001.620730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829001.620809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829001.940753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829002.452762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829002.452816:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829002.644748:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829002.644748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829002.644760:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829002.644765:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829002.644776:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2009 00000100:00100000:1.0:1787829002.644777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829002.644788:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829002.644797:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829002.644849:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518012544/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829002.644879:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829002.644883:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829002.644885:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829002.644890:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2009 00000100:00080000:0.0:1787829002.644894:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829002.644898:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829002.644909:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829002.644925:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518012672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829002.644929:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518012544:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829002.644946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829002.644950:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829002.644952:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829002.644957:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2009 00000100:00080000:0.0:1787829002.644962:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829002.644977:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829002.644983:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829002.644998:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518012800/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829002.645007:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829002.645017:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829002.645021:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829002.645023:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829002.645029:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2009 00000100:00080000:0.0:1787829002.645034:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829002.645039:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829002.645058:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829002.645068:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff8b2e880412c0 x1874672518012544/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829002.645076:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518012928/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829002.645087:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518012544/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829002.645090:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829002.645098:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829002.645102:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829002.645105:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518012544:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829002.645106:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829002.645110:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829002.645117:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2009 00000100:00080000:0.0:1787829002.645121:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829002.645127:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518012672:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829002.645129:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829002.645138:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829002.645151:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829002.645157:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518013056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829002.645166:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff8b2e88043840 x1874672518012672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829002.645177:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829002.645181:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518012672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829002.645183:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829002.645188:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829002.645197:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829002.645198:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2009 00000100:00080000:3.0:1787829002.645203:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829002.645206:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829002.645207:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518012672:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829002.645216:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829002.645217:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518012800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829002.645234:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829002.645241:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff8b2e88040780 x1874672518012800/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829002.645242:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829002.645255:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518012800/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829002.645263:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829002.645265:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518013184/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829002.645268:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829002.645275:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518012800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829002.645286:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518012928:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829002.645292:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829002.645298:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829002.645300:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829002.645306:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829002.645310:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2009 00000100:00080000:0.0:1787829002.645317:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829002.645322:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829002.645327:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff8b2e88042d00 x1874672518012928/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829002.645330:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829002.645349:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518013312/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829002.645351:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518012928/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829002.645378:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829002.645382:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829002.645385:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518012928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829002.645400:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518013056:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829002.645417:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829002.645425:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff8b2e880421c0 x1874672518013056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829002.645433:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518013056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829002.645439:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829002.645441:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829002.645443:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518013056:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829002.645446:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518013184:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829002.645456:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829002.645462:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff8b2e88040b40 x1874672518013184/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829002.645469:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518013184/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829002.645473:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829002.645475:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829002.645478:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518013184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829002.645481:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518013312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829002.645490:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829002.645496:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829002/real 1787829002] req@ffff8b2e88041e00 x1874672518013312/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829002.645502:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518013312/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829027 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829002.645508:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829002.645509:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829002.645511:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518013312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829002.964749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829003.476771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829003.476820:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829003.668801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829003.668819:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829003.988733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829004.500770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829004.500831:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829004.692725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829004.692726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829005.012734:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829005.524723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829005.524795:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829005.716728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829005.716728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829006.036724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829006.548776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829006.548806:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829006.740728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829006.740728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829007.060755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829007.572768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829007.572809:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829007.764787:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829007.764793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829007.764793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829007.764797:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829007.764805:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829007.764817:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2014 00000100:00080000:3.0:1787829007.764830:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829007.764836:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829007.764871:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518013440/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829007.764896:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829007.764899:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829007.764901:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829007.764904:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2014 00000100:00080000:3.0:1787829007.764906:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829007.764909:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829007.764918:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829007.764927:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518013568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829007.764933:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518013440:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829007.764938:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829007.764942:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829007.764943:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829007.764947:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2014 00000100:00080000:3.0:1787829007.764949:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829007.764951:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829007.764954:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829007.764962:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518013696/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829007.764970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829007.764973:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829007.764975:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829007.764979:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2014 00000100:00080000:3.0:1787829007.764983:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829007.764987:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787829007.764991:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:3.0:1787829007.764996:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829007.765028:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518013824/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829007.765040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829007.765043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787829007.765046:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff8b2e8803b480 x1874672518013440/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829007.765047:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829007.765058:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2014 00000100:00080000:3.0:1787829007.765064:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829007.765071:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518013440/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829007.765072:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829007.765082:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829007.765090:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829007.765098:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829007.765099:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518013952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829007.765106:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518013440:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829007.765111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829007.765115:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829007.765117:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829007.765120:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2014 00000100:00080000:3.0:1787829007.765123:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829007.765126:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829007.765129:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829007.765135:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518013568:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829007.765139:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518014080/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829007.765147:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829007.765157:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829007.765161:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829007.765162:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829007.765169:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2014 00000100:00080000:3.0:1787829007.765172:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829007.765175:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829007.765179:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff8b2e88039680 x1874672518013568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829007.765180:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829007.765195:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518014208/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829007.765197:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518013568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829007.765222:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829007.765225:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829007.765228:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518013568:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829007.765234:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518013696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829007.765247:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829007.765254:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff8b2e880392c0 x1874672518013696/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829007.765261:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518013696/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829007.765266:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829007.765268:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829007.765270:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518013696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829007.765273:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518013824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829007.765283:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829007.765288:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff8b2e88039e00 x1874672518013824/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829007.765295:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518013824/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829007.765300:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829007.765302:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829007.765304:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518013824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829007.765307:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518013952:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829007.765316:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829007.765321:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff8b2e88039a40 x1874672518013952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829007.765329:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518013952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829007.765334:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829007.765336:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829007.765337:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518013952:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829007.765348:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518014080:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829007.765358:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829007.765364:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff8b2e880383c0 x1874672518014080/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829007.765372:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518014080/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829007.765377:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829007.765378:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829007.765380:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518014080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829007.765383:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518014208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829007.765392:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829007.765398:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829007/real 1787829007] req@ffff8b2e8803a1c0 x1874672518014208/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829007.765404:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518014208/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829032 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829007.765410:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829007.765412:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829007.765413:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518014208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829008.084767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829008.596730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829008.596838:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829008.788717:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829008.788756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829009.108754:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829009.620744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829009.620771:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829009.812749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829009.812751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829010.132776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829010.644725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829010.644757:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829010.836727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829010.836745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829011.156724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829011.668738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829011.668812:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829011.860792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829011.860802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829012.180765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829012.692733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829012.692758:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829012.884800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829012.884810:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829012.884823:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829012.884829:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829012.884840:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2019 00000100:00080000:3.0:1787829012.884854:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829012.884864:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829012.884881:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829012.884913:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518014336/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829012.884946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829012.884950:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829012.884952:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829012.884957:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2019 00000100:00080000:3.0:1787829012.884962:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829012.884966:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829012.885004:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829012.885018:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518014336:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829012.885026:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518014464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829012.885043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829012.885047:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829012.885050:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829012.885054:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2019 00000100:00080000:3.0:1787829012.885060:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829012.885064:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829012.885071:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829012.885088:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518014592/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829012.885099:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829012.885104:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829012.885107:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829012.885112:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829012.885118:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2019 00000100:00080000:3.0:1787829012.885125:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829012.885129:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829012.885135:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829012.885149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518014720/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829012.885164:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff8b2e8803b0c0 x1874672518014336/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829012.885172:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829012.885184:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829012.885190:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829012.885198:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2019 00000100:00080000:3.0:1787829012.885204:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829012.885212:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518014336/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829012.885213:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829012.885222:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829012.885241:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518014848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829012.885251:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829012.885256:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829012.885261:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829012.885266:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829012.885267:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829012.885283:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2019 00000100:00100000:0.0:1787829012.885285:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518014336:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829012.885287:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829012.885297:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829012.885306:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829012.885317:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518014976/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829012.885331:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829012.885338:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518014464:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829012.885339:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829012.885345:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829012.885361:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2019 00000100:00080000:3.0:1787829012.885368:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829012.885372:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829012.885377:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829012.885390:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518015104/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829012.885403:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829012.885437:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff8b2e8803b840 x1874672518014464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829012.885464:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518014464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829012.885478:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829012.885483:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829012.885497:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518014464:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829012.885505:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518014592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829012.885533:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829012.885547:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff8b2e88038f00 x1874672518014592/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829012.885559:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518014592/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829012.885569:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829012.885572:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829012.885576:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518014592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829012.885582:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518014720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829012.885598:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829012.885608:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff8b2e8803a580 x1874672518014720/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829012.885619:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518014720/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829012.885628:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829012.885631:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829012.885635:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518014720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829012.885640:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518014848:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829012.885701:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829012.885721:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff8b2e8803a1c0 x1874672518014848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829012.885743:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518014848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829012.885759:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829012.885765:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829012.885770:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518014848:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829012.885797:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518014976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829012.885823:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829012.885835:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff8b2e880383c0 x1874672518014976/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829012.885848:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518014976/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829012.885857:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829012.885861:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829012.885864:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518014976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829012.885870:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518015104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829012.885886:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829012.885895:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829012/real 1787829012] req@ffff8b2e88039a40 x1874672518015104/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829012.885916:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518015104/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829037 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829012.885924:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829012.885927:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829012.885930:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518015104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829013.204750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829013.716720:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829013.716761:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829013.908826:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829013.908826:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829014.228756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829014.740753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829014.740858:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829014.932818:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829014.932818:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829015.252757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829015.764771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829015.764813:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829015.956780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829015.956780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829016.276766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829016.788712:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829016.788775:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829016.980740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829016.980740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829017.300761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829017.812771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829017.812849:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829018.004779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829018.004779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829018.004834:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829018.004846:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829018.004853:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829018.004868:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2024 00000100:00080000:3.0:1787829018.004904:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829018.004916:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829018.005027:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518015232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829018.005065:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829018.005070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829018.005073:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829018.005080:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2024 00000100:00080000:3.0:1787829018.005086:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829018.005091:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829018.005106:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518015232:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787829018.005107:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829018.005135:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518015360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829018.005151:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829018.005155:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829018.005157:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829018.005163:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2024 00000100:00080000:3.0:1787829018.005168:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829018.005173:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829018.005180:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1787829018.005207:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829018.005216:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518015488/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829018.005229:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829018.005234:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829018.005238:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829018.005242:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2024 00000100:00080000:3.0:1787829018.005248:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829018.005255:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829018.005265:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829018.005282:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829018/real 1787829018] req@ffff8b2e88039e00 x1874672518015232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829018.005294:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518015616/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829018.005338:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518015232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829018.005339:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829018.005365:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829018.005377:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829018.005378:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829018.005395:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829018.005403:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2024 00000100:00100000:0.0:1787829018.005407:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518015232:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829018.005419:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829018.005432:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829018.005446:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518015360:192.168.203.101@tcp:38:kworker.0 00000008:00100000:3.0:1787829018.005447:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829018.005496:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518015744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829018.005513:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829018.005538:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829018/real 1787829018] req@ffff8b2e880392c0 x1874672518015360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829018.005539:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829018.005575:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518015360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829018.005590:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829018.005597:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829018.005635:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829018.005646:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2024 00000100:00080000:0.0:1787829018.005651:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829018.005710:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518015360:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829018.005749:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518015488:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787829018.005761:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829018.005772:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829018.005780:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829018.005788:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829018/real 1787829018] req@ffff8b2e88039680 x1874672518015488/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829018.005793:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829018.005815:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518015488/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829018.005832:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829018.005835:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829018.005839:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518015488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829018.005843:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518015872/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829018.005853:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518015616:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787829018.005899:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829018.005906:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829018.005911:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829018.005920:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2024 00000800:00000100:0.0:1787829018.005932:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829018.005971:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829018.006011:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829018.006018:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829018/real 1787829018] req@ffff8b2e8803b480 x1874672518015616/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829018.006020:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829018.006042:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518015616/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829018.006050:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518016000/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829018.006076:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829018.006087:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829018.006096:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518015616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829018.006117:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518015744:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829018.006133:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829018.006140:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829018/real 1787829018] req@ffff8b2e88039e00 x1874672518015744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829018.006148:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518015744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829018.006154:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829018.006156:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829018.006158:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518015744:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829018.006162:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518015872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829018.006171:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829018.006176:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829018/real 1787829018] req@ffff8b2e88039a40 x1874672518015872/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829018.006183:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518015872/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829018.006188:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829018.006190:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829018.006192:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518015872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829018.006196:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518016000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829018.006205:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829018.006211:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829018/real 1787829018] req@ffff8b2e880383c0 x1874672518016000/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829018.006218:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518016000/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829043 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829018.006223:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829018.006225:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829018.006227:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518016000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829018.324770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829018.836809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829018.836863:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829019.028729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829019.028730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829019.348749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829019.860789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829019.860829:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829020.052736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829020.052763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829020.372794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829020.884801:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829020.884854:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829021.076777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829021.076812:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829021.396769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829021.908754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829021.908795:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829022.100750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829022.100752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829022.420764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829022.932728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829022.932779:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829023.124727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829023.124732:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829023.124739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829023.124740:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829023.124746:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829023.124756:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2029 00000100:00080000:3.0:1787829023.124765:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829023.124772:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829023.124806:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518016128/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829023.124826:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829023.124829:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829023.124831:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829023.124834:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2029 00000100:00080000:3.0:1787829023.124836:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829023.124839:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829023.124848:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829023.124857:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518016256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829023.124867:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518016128:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829023.124873:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829023.124878:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829023.124879:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829023.124883:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2029 00000100:00080000:3.0:1787829023.124885:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829023.124887:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829023.124890:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829023.124900:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518016384/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829023.124920:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829023.124940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829023.124942:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829023.124946:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2029 00000100:00080000:3.0:1787829023.124951:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829023.124963:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829023.124969:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829023.124983:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829023.124989:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518016512/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829023.125009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829023.125014:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829023.125017:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829023.125023:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2029 00000100:00080000:3.0:1787829023.125032:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829023.125045:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829023/real 1787829023] req@ffff8b2e8803a1c0 x1874672518016128/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829023.125058:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829023.125083:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829023.125093:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518016128/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829023.125108:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518016640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829023.125113:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829023.125127:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829023.125133:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829023.125140:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518016128:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829023.125141:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829023.125147:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829023.125171:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2029 00000100:00080000:3.0:1787829023.125174:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829023.125177:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829023.125181:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829023.125191:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518016768/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829023.125194:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518016256:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829023.125208:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829023.125211:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829023.125212:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829023.125233:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2029 00000100:00080000:3.0:1787829023.125236:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829023.125239:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829023.125244:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829023.125258:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518016896/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829023.125270:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829023.125303:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829023/real 1787829023] req@ffff8b2e8803a580 x1874672518016256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829023.125327:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518016256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829023.125356:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829023.125360:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829023.125363:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518016256:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829023.125368:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518016384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829023.125390:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829023.125419:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829023/real 1787829023] req@ffff8b2e88038f00 x1874672518016384/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829023.125438:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518016384/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829023.125448:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829023.125453:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829023.125458:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518016384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829023.125468:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518016512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829023.125498:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829023.125515:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829023/real 1787829023] req@ffff8b2e8803b840 x1874672518016512/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829023.125533:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518016512/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829023.125561:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829023.125564:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829023.125568:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518016512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829023.125574:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518016640:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829023.125590:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829023.125601:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829023/real 1787829023] req@ffff8b2e8803b0c0 x1874672518016640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829023.125611:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518016640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829023.125620:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829023.125623:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829023.125626:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518016640:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829023.125642:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518016768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829023.125659:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829023.125670:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829023/real 1787829023] req@ffff8b2e8803a1c0 x1874672518016768/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829023.125680:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518016768/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829023.125688:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829023.125711:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829023.125716:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518016768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829023.125723:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518016896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829023.125740:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829023.125752:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829023/real 1787829023] req@ffff8b2e880383c0 x1874672518016896/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829023.125764:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518016896/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829048 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829023.125773:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829023.125776:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829023.125778:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518016896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829023.444720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829023.956761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829023.956828:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829024.148721:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829024.148727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829024.468776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829024.980750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829024.980851:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829025.172859:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829025.172866:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829025.492731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829026.004732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829026.004772:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829026.196742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829026.196744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829026.516773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829027.028726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829027.028809:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829027.220765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829027.220780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829027.540712:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829028.052759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829028.052792:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829028.244742:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829028.244749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829028.244749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829028.244754:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829028.244761:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829028.244772:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2034 00000100:00080000:3.0:1787829028.244784:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829028.244792:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829028.244830:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518017024/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829028.244857:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829028.244861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829028.244863:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829028.244879:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2034 00000100:00080000:3.0:1787829028.244884:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829028.244888:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829028.244900:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829028.244906:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518017024:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829028.244918:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518017152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829028.244931:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829028.244936:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829028.244938:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829028.244942:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2034 00000100:00080000:3.0:1787829028.244947:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829028.244951:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829028.244957:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1787829028.244964:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829028.244970:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518017280/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829028.244982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829028.244986:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829028.244988:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829028.244992:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2034 00000100:00080000:3.0:1787829028.244996:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829028.245000:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829028.245005:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829028.245008:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff8b2e88039a40 x1874672518017024/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829028.245019:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518017408/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829028.245032:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829028.245036:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518017024/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829028.245037:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829028.245040:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829028.245049:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2034 00000100:00080000:3.0:1787829028.245056:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829028.245059:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829028.245062:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829028.245068:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:3.0:1787829028.245070:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829028.245073:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518017024:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829028.245086:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518017536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829028.245097:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518017152:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829028.245101:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829028.245106:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829028.245109:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829028.245114:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2034 00000100:00080000:3.0:1787829028.245119:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829028.245123:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829028.245128:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829028.245134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829028.245144:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff8b2e88039e00 x1874672518017152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829028.245151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518017664/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829028.245168:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518017152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829028.245168:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829028.245177:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829028.245181:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829028.245190:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2034 00000100:00080000:3.0:1787829028.245196:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829028.245202:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829028.245203:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829028.245208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829028.245210:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829028.245217:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518017152:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829028.245223:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518017792/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829028.245229:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518017280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829028.245262:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829028.245274:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff8b2e8803b480 x1874672518017280/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829028.245286:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518017280/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829028.245294:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829028.245297:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829028.245300:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518017280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829028.245306:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518017408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829028.245324:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829028.245336:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff8b2e88039680 x1874672518017408/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829028.245347:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518017408/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829028.245353:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829028.245355:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829028.245357:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518017408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829028.245370:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518017536:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829028.245384:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829028.245391:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff8b2e880392c0 x1874672518017536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829028.245398:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518017536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829028.245404:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829028.245406:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829028.245408:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518017536:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829028.245411:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518017664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829028.245421:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829028.245426:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff8b2e88039a40 x1874672518017664/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829028.245433:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518017664/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829028.245438:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829028.245440:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829028.245442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518017664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829028.245445:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518017792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829028.245453:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829028.245459:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829028/real 1787829028] req@ffff8b2e880383c0 x1874672518017792/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829028.245466:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518017792/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829053 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829028.245471:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829028.245473:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829028.245475:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518017792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829028.564765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829029.076725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829029.076776:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829029.268733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829029.268736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829029.588820:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829030.100743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829030.100775:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829030.292720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829030.292731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829030.612809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829031.124783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829031.124840:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829031.316776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829031.316845:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829031.636746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829032.148726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829032.148783:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829032.340764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829032.340764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829032.660790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829033.172813:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829033.172884:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:1.0:1787829033.364783:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829033.364784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829033.364784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829033.364807:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829033.364811:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.364820:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2039 00000100:00080000:1.0:1787829033.364830:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.364848:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829033.364897:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518017920/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829033.364919:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.364922:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829033.364924:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.364927:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2039 00000100:00080000:1.0:1787829033.364930:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.364933:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829033.364941:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829033.364950:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518017920:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829033.364955:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518018048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829033.364974:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.364977:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829033.364978:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.364981:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2039 00000100:00080000:1.0:1787829033.364983:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.364985:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829033.364989:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829033.364998:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518018176/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829033.365004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.365007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829033.365008:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.365010:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2039 00000100:00080000:1.0:1787829033.365012:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.365015:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787829033.365019:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787829033.365020:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829033.365030:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518018304/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829033.365037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.365040:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829033.365041:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.365044:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2039 00000100:00080000:1.0:1787829033.365046:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.365048:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829033.365052:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829033.365060:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518018432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829033.365065:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829033] req@ffff8b2e88077c00 x1874672518017920/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829033.365073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.365081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829033.365083:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.365087:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2039 00000100:00080000:1.0:1787829033.365090:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829033.365091:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518017920/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829033.365096:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829033.365103:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829033.365108:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829033.365113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518018560/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829033.365116:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829033.365124:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518017920:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829033.365127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829033.365131:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829033.365132:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829033.365136:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2039 00000100:00080000:1.0:1787829033.365138:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829033.365140:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829033.365142:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829033.365148:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518018688/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829033.365156:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518018048:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829033.365186:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829033.365199:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829033] req@ffff8b2e88077480 x1874672518018048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829033.365207:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518018048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829033.365218:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829033.365220:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829033.365223:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518018048:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829033.365227:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518018176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829033.365245:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829033.365252:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829033] req@ffff8b2e88075a40 x1874672518018176/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829033.365260:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518018176/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829033.365265:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829033.365267:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829033.365269:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518018176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829033.365273:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518018304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829033.365289:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829033.365294:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829033] req@ffff8b2e88075680 x1874672518018304/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829033.365301:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518018304/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829033.365306:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829033.365308:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829033.365311:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518018304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829033.365314:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518018432:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829033.365324:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829033.365330:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829033] req@ffff8b2e88074f00 x1874672518018432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829033.365337:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518018432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829033.365345:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829033.365348:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829033.365351:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518018432:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829033.365357:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518018560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829033.365389:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829033.365401:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829033] req@ffff8b2e88076940 x1874672518018560/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829033.365413:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518018560/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829033.365419:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829033.365421:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829033.365424:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518018560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829033.365438:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518018688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829033.365453:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829033.365461:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829033/real 1787829033] req@ffff8b2e88074780 x1874672518018688/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829033.365473:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518018688/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829058 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829033.365482:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829033.365485:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829033.365488:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518018688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829033.684707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829034.196750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829034.196867:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829034.388791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829034.388817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829034.708735:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829035.220726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829035.220746:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829035.412752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829035.412752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829035.732707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829036.244706:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829036.244769:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829036.436759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829036.436759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829036.756747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829037.268737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829037.268815:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829037.460769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829037.460769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829037.780713:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829038.292741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829038.292772:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829038.484728:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.484737:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787829038.484739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829038.484739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829038.484742:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829038.484754:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2044 00000100:00080000:1.0:1787829038.484766:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.484773:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829038.484829:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518018816/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829038.484853:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.484856:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829038.484857:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829038.484860:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2044 00000100:00080000:1.0:1787829038.484864:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.484867:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829038.484877:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829038.484887:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518018816:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829038.484889:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518018944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829038.484900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.484903:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829038.484906:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829038.484908:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2044 00000100:00080000:1.0:1787829038.484911:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.484913:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829038.484917:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829038.484925:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518019072/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829038.484931:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.484934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829038.484935:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829038.484938:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2044 00000100:00080000:1.0:1787829038.484957:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.484961:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829038.484967:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829038.484969:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829038.484985:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518019200/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829038.484998:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.485001:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829038.485004:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829038.485009:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2044 00000100:00080000:1.0:1787829038.485013:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.485018:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829038.485025:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829038.485026:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff8b2e88077840 x1874672518018816/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829038.485043:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518019328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829038.485053:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518018816/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829038.485058:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.485067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829038.485069:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829038.485070:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829038.485076:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829038.485081:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2044 00000100:00100000:0.0:1787829038.485082:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518018816:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829038.485101:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.485109:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829038.485112:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518018944:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787829038.485117:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829038.485132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518019456/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829038.485141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829038.485144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787829038.485148:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829038.485148:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829038.485155:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2044 00000100:00080000:1.0:1787829038.485160:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829038.485163:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829038.485166:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff8b2e88076580 x1874672518018944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829038.485169:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829038.485185:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518019584/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829038.485186:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518018944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829038.485207:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829038.485210:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829038.485213:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518018944:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829038.485218:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518019072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829038.485232:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829038.485240:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff8b2e88076d00 x1874672518019072/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829038.485248:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518019072/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829038.485254:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829038.485256:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829038.485258:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518019072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829038.485261:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518019200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829038.485271:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829038.485277:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff8b2e880770c0 x1874672518019200/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829038.485284:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518019200/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829038.485289:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829038.485291:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829038.485294:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518019200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829038.485298:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518019328:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829038.485308:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829038.485313:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff8b2e88074000 x1874672518019328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829038.485321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518019328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829038.485326:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829038.485328:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829038.485330:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518019328:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829038.485347:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518019456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829038.485358:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829038.485365:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff8b2e880761c0 x1874672518019456/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829038.485393:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518019456/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829038.485403:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829038.485407:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829038.485410:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518019456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829038.485418:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518019584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829038.485442:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829038.485454:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829038/real 1787829038] req@ffff8b2e88075e00 x1874672518019584/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829038.485479:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518019584/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829063 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829038.485485:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829038.485487:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829038.485489:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518019584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829038.804765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829039.316759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829039.316858:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829039.508782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829039.508784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829039.828707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829040.340765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829040.340830:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829040.532801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829040.532805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829040.852730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829041.364723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829041.364802:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829041.556761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829041.556761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829041.876785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829042.388728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829042.388777:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829042.580724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829042.580724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829042.900752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829043.412795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829043.412859:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829043.604751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829043.604771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829043.604800:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.604806:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829043.604811:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.604821:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2050 00000100:00080000:0.0:1787829043.604834:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.604842:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829043.604887:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518019712/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829043.604916:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.604919:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829043.604920:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.604924:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2050 00000100:00080000:0.0:1787829043.604926:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.604929:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829043.604939:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829043.604950:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518019840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829043.604960:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.604963:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829043.604966:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829043.604967:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518019712:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829043.604972:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2050 00000100:00080000:0.0:1787829043.604976:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.604979:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829043.604982:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829043.604990:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518019968/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829043.604996:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.604999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829043.605000:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.605003:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2050 00000100:00080000:0.0:1787829043.605006:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.605008:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829043.605011:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829043.605019:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518020096/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829043.605024:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.605027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829043.605029:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.605032:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2050 00000100:00080000:0.0:1787829043.605035:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.605038:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829043.605042:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829043.605044:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829043.605053:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518020224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829043.605062:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.605065:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829043.605074:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.605078:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2050 00000100:00080000:0.0:1787829043.605081:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.605084:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829043.605088:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829043.605096:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff8b2e88042580 x1874672518019712/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829043.605100:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518020352/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829043.605112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829043.605116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829043.605119:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829043.605125:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2050 00000100:00080000:0.0:1787829043.605128:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829043.605131:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829043.605133:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829043.605136:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518019712/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829043.605143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518020480/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829043.605152:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829043.605162:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829043.605168:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518019712:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829043.605192:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518019840:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829043.605211:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829043.605221:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff8b2e88041e00 x1874672518019840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829043.605230:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518019840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829043.605237:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829043.605239:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829043.605241:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518019840:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829043.605245:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518019968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829043.605257:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829043.605263:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff8b2e88040b40 x1874672518019968/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829043.605271:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518019968/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829043.605281:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829043.605284:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829043.605286:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518019968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829043.605289:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518020096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829043.605300:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829043.605306:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff8b2e880421c0 x1874672518020096/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829043.605312:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518020096/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829043.605318:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829043.605320:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829043.605321:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518020096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829043.605325:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518020224:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829043.605336:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829043.605342:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff8b2e88042d00 x1874672518020224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829043.605350:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518020224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829043.605355:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829043.605357:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829043.605360:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518020224:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829043.605364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518020352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829043.605374:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829043.605380:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff8b2e88040780 x1874672518020352/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829043.605387:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518020352/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829043.605393:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829043.605394:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829043.605396:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518020352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829043.605400:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518020480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829043.605410:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829043.605416:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829043/real 1787829043] req@ffff8b2e88043840 x1874672518020480/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829043.605424:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518020480/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829068 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829043.605429:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829043.605431:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829043.605433:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518020480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829043.924757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829044.436768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829044.436830:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829044.628725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829044.628728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829044.948716:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829045.460793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829045.460858:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829045.652723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829045.652744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829045.972791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829046.484763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829046.484829:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829046.676759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829046.676762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829046.996717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829047.508765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829047.508843:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829047.700713:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829047.700729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829048.020784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829048.532771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829048.532890:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829048.724702:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829048.724715:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829048.724750:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829048.724757:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829048.724762:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.724772:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2055 00000100:00080000:0.0:1787829048.724784:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829048.724790:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829048.724825:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518020608/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829048.724846:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829048.724849:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829048.724850:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.724853:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2055 00000100:00080000:0.0:1787829048.724855:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829048.724858:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829048.724866:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829048.724877:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518020736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829048.724885:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829048.724889:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829048.724891:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518020608:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829048.724892:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.724898:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2055 00000100:00080000:0.0:1787829048.724903:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829048.724906:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829048.724910:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829048.724918:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518020864/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829048.724931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829048.724934:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829048.724935:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.724938:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2055 00000100:00080000:0.0:1787829048.724941:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829048.724943:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829048.724946:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829048.724954:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518020992/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829048.724960:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829048.724965:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829048.724966:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829048.724969:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.724974:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2055 00000100:00080000:0.0:1787829048.724976:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829048.724979:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829048.724982:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829048.724989:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518021120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829048.724995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829048.724997:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829048.724999:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.725001:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2055 00000100:00080000:0.0:1787829048.725004:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829048.725006:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829048.725009:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829048.725017:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518021248/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829048.725025:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829048.725025:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff8b2e880412c0 x1874672518020608/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829048.725031:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829048.725033:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829048.725036:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2055 00000100:00080000:0.0:1787829048.725038:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829048.725040:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829048.725044:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829048.725053:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518021376/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829048.725069:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518020608/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829048.725084:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829048.725087:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829048.725090:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518020608:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829048.725108:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518020736:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829048.725133:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829048.725144:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff8b2e88040f00 x1874672518020736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829048.725153:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518020736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829048.725159:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829048.725162:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829048.725164:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518020736:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829048.725169:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518020864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829048.725181:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829048.725188:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff8b2e88041a40 x1874672518020864/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829048.725195:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518020864/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829048.725200:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829048.725202:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829048.725204:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518020864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829048.725207:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518020992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829048.725217:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829048.725222:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff8b2e88040000 x1874672518020992/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829048.725230:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518020992/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829048.725235:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829048.725237:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829048.725238:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518020992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829048.725242:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518021120:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829048.725251:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829048.725258:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff8b2e880430c0 x1874672518021120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829048.725265:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518021120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829048.725271:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829048.725272:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829048.725274:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518021120:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829048.725278:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518021248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829048.725287:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829048.725293:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff8b2e88043480 x1874672518021248/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829048.725306:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518021248/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829048.725314:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829048.725317:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829048.725320:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518021248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829048.725326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518021376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829048.725348:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829048.725361:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829048/real 1787829048] req@ffff8b2e88042940 x1874672518021376/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829048.725373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518021376/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829073 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829048.725381:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829048.725383:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829048.725385:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518021376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829049.044775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829049.556768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829049.556868:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829049.748734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829049.748740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829050.068773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829050.580756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829050.580813:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829050.772770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829050.772770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829051.092762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829051.604737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829051.604791:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829051.796766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829051.796766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829052.116767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829052.628780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829052.628871:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829052.820767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829052.820767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829053.140764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829053.652743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829053.652913:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829053.844755:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829053.844755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829053.844755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829053.844765:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829053.844770:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.844781:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2060 00000100:00080000:0.0:1787829053.844793:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.844801:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829053.844842:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518021504/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829053.844877:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.844881:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829053.844884:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.844890:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2060 00000100:00080000:0.0:1787829053.844895:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.844900:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829053.844906:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518021504:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829053.844912:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829053.844931:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518021632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829053.844944:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.844949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829053.844951:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.844955:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2060 00000100:00080000:0.0:1787829053.844960:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.844963:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829053.844968:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829053.844978:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518021760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829053.844985:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.844988:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829053.844989:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829053.844994:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829053.844995:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2060 00000100:00080000:0.0:1787829053.845000:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.845003:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829053.845007:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829053.845015:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518021888/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829053.845021:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.845023:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829053.845025:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.845027:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2060 00000100:00080000:0.0:1787829053.845029:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.845033:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829053.845036:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829053.845043:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff8b2e880403c0 x1874672518021504/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829053.845046:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518022016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829053.845073:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.845076:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829053.845078:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.845083:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2060 00000100:00100000:3.0:1787829053.845084:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518021504/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829053.845085:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.845091:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829053.845097:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829053.845104:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829053.845107:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518022144/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829053.845111:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829053.845119:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518021504:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829053.845131:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829053.845134:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829053.845135:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829053.845138:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2060 00000100:00080000:0.0:1787829053.845140:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829053.845143:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829053.845146:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829053.845153:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518022272/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829053.845156:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518021632:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829053.845183:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829053.845195:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff8b2e88042940 x1874672518021632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829053.845204:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518021632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829053.845210:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829053.845213:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829053.845215:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518021632:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829053.845219:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518021760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829053.845232:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829053.845238:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff8b2e88043480 x1874672518021760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829053.845246:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518021760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829053.845252:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829053.845253:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829053.845255:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518021760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829053.845259:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518021888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829053.845269:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829053.845275:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff8b2e880430c0 x1874672518021888/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829053.845282:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518021888/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829053.845288:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829053.845290:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829053.845292:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518021888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829053.845296:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518022016:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829053.845305:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829053.845311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff8b2e88040000 x1874672518022016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829053.845318:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518022016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829053.845324:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829053.845326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829053.845328:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518022016:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829053.845332:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518022144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829053.845341:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829053.845347:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff8b2e88041a40 x1874672518022144/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829053.845353:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518022144/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829053.845358:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829053.845361:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829053.845363:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518022144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829053.845374:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518022272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829053.845384:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829053.845390:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829053/real 1787829053] req@ffff8b2e88040f00 x1874672518022272/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829053.845396:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518022272/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829078 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829053.845408:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829053.845410:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829053.845412:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518022272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829054.164793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829054.676794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829054.676861:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829054.868772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829054.868772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829055.188764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829055.700725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829055.700808:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829055.892728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829055.892728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829056.212719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829056.724730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829056.724802:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829056.916771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829056.916781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829057.236733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829057.748768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829057.748798:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829057.940754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829057.940754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829058.260760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829058.772778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829058.772817:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829058.964749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829058.964749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829058.964759:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829058.964766:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829058.964770:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829058.964781:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2065 00000100:00080000:0.0:1787829058.964793:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829058.964800:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829058.964833:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518022400/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829058.964855:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829058.964858:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829058.964860:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829058.964862:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2065 00000100:00080000:0.0:1787829058.964865:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829058.964867:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829058.964877:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829058.964886:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518022528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829058.964894:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518022400:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829058.964895:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829058.964900:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829058.964904:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829058.964908:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2065 00000100:00080000:0.0:1787829058.964909:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829058.964911:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829058.964914:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829058.964924:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518022656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829058.964930:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829058.964931:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829058.964932:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829058.964934:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2065 00000100:00080000:0.0:1787829058.964936:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829058.964939:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829058.964942:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829058.964950:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518022784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829058.964956:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829058.964958:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829058.964960:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829058.964962:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2065 00000100:00080000:0.0:1787829058.964972:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829058.964975:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829058.964978:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829058.964986:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518022912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829058.964992:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829058.964994:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829058.964995:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829058.964999:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2065 00000100:00080000:0.0:1787829058.965001:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829058.965004:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829058.965008:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829058.965015:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518023040/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829058.965022:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829058.965023:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518022528:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829058.965027:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829058.965031:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829058.965035:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2065 00000100:00080000:0.0:1787829058.965038:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829058.965040:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829058.965044:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829058.965060:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518023168/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829058.965064:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518022656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829058.965100:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518022784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829058.965124:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518022912:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829058.965141:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518023040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829058.965153:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518023168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829059.284751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829059.796751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829059.796822:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829059.988711:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829059.988817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829060.308737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829060.820769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829060.820812:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829061.012755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829061.012853:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829061.332776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829061.844746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829061.844814:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829062.036714:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829062.036937:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:0.0:1787829062.036998:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:0.0:1787829062.037038:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829062.037058:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829062.037108:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829062.037125:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829062.037137:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829062.037144:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829062] req@ffff8b2e88043840 x1874672518022528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829062.037153:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829062.037179:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829062.037183:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518022528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829062.037196:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00080000:3.0:1787829062.037204:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829062.037213:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829062.037218:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518022528:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829062.037228:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829062] req@ffff8b2e88040780 x1874672518022656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829062.037243:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518022656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829062.037251:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829062.037255:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829062.037259:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518022656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829062.037267:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829062] req@ffff8b2e88042d00 x1874672518022784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829062.037279:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518022784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829062.037286:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829062.037290:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829062.037293:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518022784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829062.037301:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829062] req@ffff8b2e880421c0 x1874672518022912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829062.037314:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518022912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829062.037324:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829062.037328:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829062.037332:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518022912:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829062.037337:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829062] req@ffff8b2e880412c0 x1874672518022400/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829062.037358:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518022400/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829062.037368:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829062.037371:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829062.037375:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518022400:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829062.037407:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829062] req@ffff8b2e88040b40 x1874672518023040/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829062.037420:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518023040/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829062.037429:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829062.037442:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829062.037446:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518023040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829062.037453:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829058/real 1787829062] req@ffff8b2e88041e00 x1874672518023168/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829062.037467:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518023168/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829083 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829062.037476:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829062.037479:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829062.037483:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518023168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829062.356715:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829062.868764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829062.868849:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829063.060734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829063.060742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829063.380763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829063.892750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829063.892807:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829064.084772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829064.084778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829064.084778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829064.084783:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829064.084790:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829064.084799:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2070 00000100:00080000:0.0:1787829064.084808:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829064.084815:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829064.084848:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518023296/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829064.084870:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829064.084873:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829064.084874:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829064.084877:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2070 00000100:00080000:0.0:1787829064.084879:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829064.084882:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829064.084890:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829064.084911:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518023424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829064.084921:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518023296:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829064.084922:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829064.084927:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829064.084929:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829064.084935:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2070 00000100:00080000:0.0:1787829064.084937:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829064.084943:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829064.084947:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829064.084955:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518023552/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829064.084961:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829064.084964:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829064.084965:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829064.084967:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2070 00000100:00080000:0.0:1787829064.084969:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829064.084971:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829064.084974:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829064.084983:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518023680/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829064.084992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829064.084995:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829064.084997:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829064.085002:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2070 00000800:00000100:3.0:1787829064.085008:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829064.085010:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829064.085017:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829064.085024:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829064.085037:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518023808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829064.085047:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829064.085049:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829064.085051:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829064.085055:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2070 00000100:00080000:0.0:1787829064.085060:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829064.085061:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829064/real 1787829064] req@ffff8b2e88042580 x1874672518023296/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829064.085068:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829064.085078:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829064.085096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518023936/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829064.085103:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518023296/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829064.085109:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829064.085118:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829064.085122:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829064.085127:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829064.085129:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2070 00000100:00080000:0.0:1787829064.085138:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829064.085138:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829064.085148:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829064.085149:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518023296:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829064.085158:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829064.085175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518024064/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829064.085185:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518023424:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829064.085218:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829064.085241:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829064/real 1787829064] req@ffff8b2e88041e00 x1874672518023424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829064.085255:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518023424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829064.085264:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829064.085267:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829064.085271:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518023424:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829064.085277:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518023552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829064.085308:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829064.085321:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829064/real 1787829064] req@ffff8b2e88040b40 x1874672518023552/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829064.085334:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518023552/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829064.085348:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829064.085351:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829064.085354:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518023552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829064.085361:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518023680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829064.085382:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829064.085393:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829064/real 1787829064] req@ffff8b2e880412c0 x1874672518023680/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829064.085404:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518023680/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829064.085412:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829064.085415:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829064.085418:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518023680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829064.085425:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518023808:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829064.085443:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829064.085454:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829064/real 1787829064] req@ffff8b2e880421c0 x1874672518023808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829064.085465:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518023808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829064.085474:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829064.085476:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829064.085480:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518023808:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829064.085487:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518023936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829064.085508:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829064.085521:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829064/real 1787829064] req@ffff8b2e88042d00 x1874672518023936/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829064.085533:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518023936/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829064.085542:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829064.085545:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829064.085549:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518023936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829064.086426:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518024064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829064.086463:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829064.086482:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829064/real 1787829064] req@ffff8b2e88040780 x1874672518024064/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829064.086497:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518024064/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829089 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829064.086586:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829064.086589:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829064.086592:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518024064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829064.404712:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829064.916774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829064.916835:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829065.108720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829065.108720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829065.428800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829065.940747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829065.940804:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829066.132776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829066.132785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829066.452758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829066.964740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829066.964793:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829067.156734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829067.156750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829067.476755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829067.988723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829067.988829:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829068.180742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829068.180743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829068.500731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829069.012765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829069.012871:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829069.204761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829069.204765:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829069.204774:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829069.204779:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829069.204790:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2075 00000100:00080000:0.0:1787829069.204802:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829069.204810:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829069.204861:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518024192/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829069.204901:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829069.204905:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829069.204910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829069.204911:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829069.204938:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2075 00000100:00080000:0.0:1787829069.204942:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829069.204945:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829069.204956:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829069.204970:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518024320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829069.204979:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829069.204982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829069.204984:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829069.204987:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2075 00000100:00080000:0.0:1787829069.204989:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829069.204993:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829069.204998:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829069.205006:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518024448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829069.205013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829069.205016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829069.205017:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829069.205020:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2075 00000100:00080000:0.0:1787829069.205022:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829069.205024:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829069.205028:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829069.205039:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518024576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829069.205039:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518024192:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829069.205068:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829069.205070:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829069.205071:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829069.205075:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2075 00000100:00080000:0.0:1787829069.205078:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829069.205081:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829069.205086:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829069.205098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518024704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829069.205105:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829069.205108:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829069.205110:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829069.205113:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2075 00000100:00080000:0.0:1787829069.205118:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829069.205121:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829069.205125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829069.205133:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518024832/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829069.205138:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829069.205141:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829069.205142:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829069.205145:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2075 00000100:00080000:0.0:1787829069.205149:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829069.205153:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829069.205155:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829069.205164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829069.205174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518024960/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829069.205280:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829069/real 1787829069] req@ffff8b2e88043840 x1874672518024192/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829069.205308:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518024192/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829069.205322:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829069.205327:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829069.205330:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518024192:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829069.205339:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518024320:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829069.205377:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829069.205414:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829069/real 1787829069] req@ffff8b2e88040f00 x1874672518024320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829069.205431:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518024320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829069.205443:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829069.205448:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829069.205464:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518024320:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829069.205473:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518024448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829069.205497:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829069.205508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829069/real 1787829069] req@ffff8b2e88041a40 x1874672518024448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829069.205518:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518024448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829069.205524:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829069.205527:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829069.205530:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518024448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829069.205553:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518024576:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829069.205567:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829069.205574:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829069/real 1787829069] req@ffff8b2e88040000 x1874672518024576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829069.205602:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518024576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829069.205610:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829069.205613:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829069.205616:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518024576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829069.205622:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518024704:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829069.205692:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829069.205710:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829069/real 1787829069] req@ffff8b2e880430c0 x1874672518024704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829069.205720:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518024704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829069.205728:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829069.205731:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829069.205734:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518024704:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829069.205739:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518024832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829069.205753:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829069.205761:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829069/real 1787829069] req@ffff8b2e88043480 x1874672518024832/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829069.205770:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518024832/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829069.205777:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829069.205779:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829069.205782:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518024832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829069.205786:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518024960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829069.205798:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829069.205805:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829069/real 1787829069] req@ffff8b2e88042940 x1874672518024960/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829069.205823:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518024960/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829094 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829069.205830:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829069.205832:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829069.205835:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518024960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829069.524717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829070.036816:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829070.036864:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829070.228751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829070.228751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829070.548706:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829071.060728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829071.060786:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829071.252737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829071.252741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829071.572754:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829072.084996:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829072.085054:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787829072.276844:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829072.276875:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829072.596760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829073.108802:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829073.108902:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829073.300780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829073.300797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829073.620807:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829074.132764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829074.132815:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829074.324757:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829074.324758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829074.324760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829074.324768:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829074.324773:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829074.324783:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2080 00000100:00080000:0.0:1787829074.324793:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829074.324800:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829074.324836:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518025088/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829074.324858:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829074.324861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829074.324862:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829074.324865:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2080 00000100:00080000:0.0:1787829074.324868:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829074.324871:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829074.324879:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829074.324891:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518025088:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829074.324893:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518025216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829074.324914:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829074.324917:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829074.324918:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829074.324921:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2080 00000100:00080000:0.0:1787829074.324923:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829074.324926:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829074.324929:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829074.324938:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518025344/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829074.324943:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829074.324945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829074.324946:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829074.324949:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2080 00000100:00080000:0.0:1787829074.324952:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829074.324955:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829074.324958:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829074.324962:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829074.324968:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518025472/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829074.324976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829074.324978:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829074.324980:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829074.324983:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2080 00000100:00080000:0.0:1787829074.324985:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829074.324988:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829074.324990:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829074.324998:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518025600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829074.325006:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829074.325007:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff8b2e880403c0 x1874672518025088/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829074.325012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829074.325018:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829074.325026:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2080 00000100:00100000:3.0:1787829074.325030:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518025088/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829074.325033:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829074.325043:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829074.325045:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829074.325051:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:0.0:1787829074.325052:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829074.325055:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518025088:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829074.325070:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518025728/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829074.325078:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518025216:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829074.325094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829074.325097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829074.325099:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829074.325104:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2080 00000800:00000100:3.0:1787829074.325107:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829074.325110:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829074.325116:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829074.325120:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829074.325123:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff8b2e88042940 x1874672518025216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829074.325133:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518025856/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829074.325136:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518025216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829074.325160:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829074.325163:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829074.325166:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518025216:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829074.325172:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518025344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829074.325185:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829074.325192:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff8b2e88043480 x1874672518025344/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829074.325200:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518025344/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829074.325205:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829074.325207:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829074.325209:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518025344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829074.325212:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518025472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829074.325221:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829074.325227:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff8b2e880430c0 x1874672518025472/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829074.325234:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518025472/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829074.325240:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829074.325242:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829074.325244:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518025472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829074.325247:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518025600:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829074.325268:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829074.325273:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff8b2e88040000 x1874672518025600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829074.325280:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518025600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829074.325286:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829074.325288:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829074.325290:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518025600:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829074.325300:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518025728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829074.325309:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829074.325315:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff8b2e88041a40 x1874672518025728/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829074.325323:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518025728/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829074.325328:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829074.325329:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829074.325332:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518025728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829074.325336:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518025856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829074.325345:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829074.325351:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829074/real 1787829074] req@ffff8b2e88040f00 x1874672518025856/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829074.325357:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518025856/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829099 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829074.325363:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829074.325364:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829074.325366:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518025856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829074.644762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829075.156752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829075.156838:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829075.348736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829075.348742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829075.668745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829076.180762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829076.180787:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829076.372781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829076.372781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829076.692713:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829077.204733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829077.204805:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829077.396741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829077.396741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829077.716752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829078.228747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829078.228810:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829078.420744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829078.420770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829078.740729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829079.252722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829079.252787:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829079.444718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829079.444724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829079.444733:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829079.444740:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829079.444745:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829079.444755:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2085 00000100:00080000:2.0:1787829079.444766:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829079.444774:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829079.444815:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518025984/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829079.444838:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829079.444842:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829079.444844:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829079.444849:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2085 00000100:00080000:2.0:1787829079.444853:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829079.444858:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829079.444868:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518025984:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787829079.444869:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829079.444882:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518026112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829079.444892:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829079.444896:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829079.444897:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829079.444901:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2085 00000100:00080000:2.0:1787829079.444913:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829079.444917:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829079.444921:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829079.444927:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829079.444931:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518026240/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829079.444939:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829079.444942:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829079.444943:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829079.444946:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2085 00000100:00080000:2.0:1787829079.444949:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829079.444952:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829079.444956:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829079.444965:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518026368/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829079.444972:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff8b2e8801de00 x1874672518025984/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829079.444973:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829079.444977:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829079.444980:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829079.444984:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2085 00000100:00080000:2.0:1787829079.444987:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829079.444991:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829079.444994:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829079.444998:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518025984/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829079.445003:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518026496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829079.445008:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829079.445012:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829079.445013:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829079.445016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829079.445017:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518025984:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829079.445018:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829079.445021:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2085 00000100:00080000:2.0:1787829079.445024:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829079.445028:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829079.445032:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829079.445040:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518026624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829079.445048:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518026112:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829079.445049:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829079.445052:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829079.445054:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829079.445058:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2085 00000100:00080000:2.0:1787829079.445060:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829079.445064:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829079.445068:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829079.445071:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829079.445078:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518026752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829079.445084:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff8b2e8801d680 x1874672518026112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829079.445096:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518026112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829079.445105:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829079.445108:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829079.445112:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518026112:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829079.445119:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518026240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829079.445139:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829079.445151:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff8b2e8801f0c0 x1874672518026240/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829079.445164:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518026240/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829079.445180:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829079.445183:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829079.445187:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518026240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829079.445193:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518026368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829079.445214:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829079.445226:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff8b2e8801c780 x1874672518026368/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829079.445238:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518026368/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829079.445247:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829079.445250:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829079.445254:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518026368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829079.445259:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518026496:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829079.445275:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829079.445283:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff8b2e8801ed00 x1874672518026496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829079.445290:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518026496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829079.445296:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829079.445298:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829079.445301:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518026496:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829079.445313:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518026624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829079.445324:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829079.445330:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff8b2e8801cb40 x1874672518026624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829079.445337:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518026624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829079.445342:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829079.445344:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829079.445345:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518026624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829079.445349:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518026752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829079.445358:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829079.445364:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829079/real 1787829079] req@ffff8b2e8801c3c0 x1874672518026752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829079.445371:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518026752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829104 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829079.445376:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829079.445377:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829079.445379:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518026752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829079.764735:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829080.276710:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829080.276777:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829080.468769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829080.468770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829080.788788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829081.300725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829081.300786:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829081.492745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829081.492745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829081.812729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829082.324745:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829082.324802:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829082.516763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829082.516763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829082.836805:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829083.348729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829083.348767:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829083.540761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829083.540764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829083.860716:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829084.372766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829084.372841:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829084.564746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829084.564751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829084.564762:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829084.564768:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829084.564772:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829084.564780:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2090 00000100:00080000:1.0:1787829084.564796:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829084.564803:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829084.564838:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518026880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829084.564858:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829084.564861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829084.564862:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829084.564865:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2090 00000100:00080000:1.0:1787829084.564868:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829084.564872:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829084.564885:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829084.564888:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518026880:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829084.564898:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518027008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829084.564907:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829084.564910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829084.564911:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829084.564913:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2090 00000100:00080000:1.0:1787829084.564916:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829084.564918:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829084.564927:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829084.564936:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518027136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829084.564943:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829084.564946:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829084.564947:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829084.564949:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2090 00000100:00080000:1.0:1787829084.564952:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829084.564956:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829084.564958:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829084.564963:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829084.564971:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518027264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829084.564978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829084.564980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829084.564982:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829084.564985:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2090 00000100:00080000:1.0:1787829084.564989:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829084.564993:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829084.564999:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829084.565014:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518027392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829084.565014:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff8b2e88074b40 x1874672518026880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829084.565041:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829084.565045:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829084.565047:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829084.565051:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518026880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829084.565054:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2090 00000100:00080000:1.0:1787829084.565060:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829084.565067:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829084.565068:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829084.565072:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829084.565075:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829084.565079:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518026880:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829084.565085:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518027520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829084.565093:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829084.565095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829084.565096:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829084.565099:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2090 00000100:00080000:1.0:1787829084.565102:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829084.565103:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518027008:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829084.565114:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829084.565118:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829084.565130:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829084.565133:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518027648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829084.565143:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff8b2e88075e00 x1874672518027008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829084.565156:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518027008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829084.565164:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829084.565166:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829084.565170:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518027008:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829084.565174:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518027136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829084.565186:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829084.565192:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff8b2e880761c0 x1874672518027136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829084.565200:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518027136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829084.565205:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829084.565207:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829084.565209:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518027136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829084.565212:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518027264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829084.565222:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829084.565228:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff8b2e88074000 x1874672518027264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829084.565235:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518027264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829084.565240:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829084.565242:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829084.565244:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518027264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829084.565247:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518027392:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829084.565256:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829084.565262:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff8b2e880770c0 x1874672518027392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829084.565270:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518027392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829084.565275:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829084.565277:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829084.565278:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518027392:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829084.565282:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518027520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829084.565291:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829084.565297:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff8b2e88076d00 x1874672518027520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829084.565304:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518027520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829084.565309:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829084.565311:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829084.565313:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518027520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829084.565324:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518027648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829084.565334:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829084.565341:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829084/real 1787829084] req@ffff8b2e88076580 x1874672518027648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829084.565353:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518027648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829109 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829084.565362:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829084.565365:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829084.565368:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518027648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829084.884792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829085.396722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829085.396797:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829085.588781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829085.588781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829085.908754:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829086.420748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829086.420810:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829086.612715:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829086.612719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829086.932789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829087.444732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829087.444874:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829087.636812:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829087.636815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829087.956741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829088.468740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829088.468800:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829088.660738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829088.660746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829088.980757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829089.492727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829089.492779:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829089.684763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829089.684763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829089.684796:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.684801:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829089.684805:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.684813:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2096 00000100:00080000:2.0:1787829089.684823:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.684828:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829089.684862:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518027776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.684883:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.684885:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829089.684887:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.684889:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2096 00000100:00080000:2.0:1787829089.684892:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.684894:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829089.684902:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829089.684911:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518027904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.684919:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829089.684920:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518027776:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829089.684926:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829089.684929:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.684933:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2096 00000100:00080000:2.0:1787829089.684935:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.684937:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829089.684940:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829089.684948:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518028032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.684955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.684957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829089.684959:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.684961:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2096 00000100:00080000:2.0:1787829089.684976:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.684979:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829089.684982:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829089.684990:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518028160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829089.684995:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829089.685002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.685005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829089.685007:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.685010:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2096 00000100:00080000:2.0:1787829089.685012:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.685015:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829089.685019:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829089.685025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518028288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.685032:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.685034:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829089.685035:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829089.685037:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2096 00000100:00080000:2.0:1787829089.685039:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.685042:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829089.685045:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829089.685051:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff8b2e8801da40 x1874672518027776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829089.685057:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518028416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829089.685080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829089.685084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829089.685088:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829089.685089:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518027776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829089.685097:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2096 00000100:00080000:2.0:1787829089.685104:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829089.685110:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829089.685113:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829089.685115:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829089.685124:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829089.685129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518028544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829089.685131:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518027776:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829089.685160:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518027904:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829089.685181:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829089.685191:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff8b2e8801fc00 x1874672518027904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.685200:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518027904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.685206:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.685208:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829089.685210:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518027904:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829089.685214:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518028032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829089.685225:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829089.685231:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff8b2e8801f480 x1874672518028032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.685239:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518028032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.685253:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.685254:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829089.685256:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518028032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829089.685260:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518028160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829089.685269:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829089.685274:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff8b2e8801cf00 x1874672518028160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.685281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518028160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.685286:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.685288:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829089.685290:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518028160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829089.685294:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518028288:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829089.685302:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829089.685308:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff8b2e8801c000 x1874672518028288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.685323:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518028288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.685328:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.685330:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829089.685332:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518028288:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829089.685336:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518028416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829089.685345:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829089.685351:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff8b2e8801e580 x1874672518028416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.685358:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518028416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.685363:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.685364:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829089.685367:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518028416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829089.685371:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518028544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829089.685380:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829089.685385:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829089/real 1787829089] req@ffff8b2e8801e940 x1874672518028544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829089.685392:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518028544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829114 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829089.685397:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829089.685398:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829089.685400:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518028544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829090.004732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829090.516745:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829090.516802:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829090.708742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829090.708742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829091.028764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829091.540811:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829091.540864:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829091.732755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829091.732756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829092.052710:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829092.564747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829092.564812:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829092.756761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829092.756801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829093.076744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829093.588791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829093.588851:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829093.780811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829093.780811:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829094.100729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829094.612717:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829094.612786:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829094.804754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829094.804789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829094.804807:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829094.804813:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829094.804818:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829094.804828:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2101 00000100:00080000:0.0:1787829094.804840:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829094.804847:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829094.804889:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518028672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829094.804915:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829094.804918:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829094.804919:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829094.804923:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2101 00000100:00080000:0.0:1787829094.804925:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829094.804928:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829094.804938:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829094.804949:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518028800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829094.804956:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518028672:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829094.804959:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829094.804964:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829094.804966:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829094.804970:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2101 00000100:00080000:0.0:1787829094.804972:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829094.804975:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829094.804979:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829094.804988:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518028928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829094.804994:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829094.804997:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829094.804998:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829094.805001:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2101 00000100:00080000:0.0:1787829094.805003:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829094.805005:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829094.805008:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829094.805015:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518029056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829094.805020:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829094.805023:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829094.805024:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829094.805028:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2101 00000100:00080000:0.0:1787829094.805031:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829094.805034:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829094.805038:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829094.805044:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829094.805051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518029184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829094.805063:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829094.805068:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829094.805071:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829094.805075:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2101 00000100:00080000:0.0:1787829094.805080:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829094.805085:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829094.805091:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829094.805095:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff8b2e88043840 x1874672518028672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829094.805105:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518029312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829094.805113:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829094.805116:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518028672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829094.805117:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829094.805121:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829094.805132:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2101 00000100:00080000:3.0:1787829094.805135:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829094.805137:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829094.805142:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829094.805147:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829094.805148:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518028672:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829094.805157:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829094.805173:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518029440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829094.805177:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518028800:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829094.805213:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829094.805224:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff8b2e88040780 x1874672518028800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.805233:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518028800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.805240:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.805242:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829094.805244:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518028800:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829094.805249:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518028928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829094.805262:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829094.805268:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff8b2e88042d00 x1874672518028928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.805276:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518028928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.805282:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.805284:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829094.805286:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518028928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829094.805291:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518029056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829094.805301:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829094.805307:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff8b2e880421c0 x1874672518029056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.805315:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518029056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.805320:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.805322:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829094.805324:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518029056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829094.805328:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518029184:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829094.805338:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829094.805344:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff8b2e880412c0 x1874672518029184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.805351:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518029184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.805357:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.805359:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829094.805361:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518029184:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829094.805365:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518029312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829094.805374:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829094.805380:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff8b2e88040b40 x1874672518029312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.805388:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518029312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.805394:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.805395:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829094.805397:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518029312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829094.805428:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518029440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829094.805443:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829094.805451:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829094/real 1787829094] req@ffff8b2e88041e00 x1874672518029440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829094.805459:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518029440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829119 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829094.805465:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829094.805468:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829094.805470:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518029440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829095.124779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829095.636787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829095.636847:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829095.828752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829095.828752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829096.148728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829096.660735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829096.660823:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829096.852744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829096.852747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829097.172751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829097.684771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829097.684803:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829097.876729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829097.876735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829098.196761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829098.708764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829098.708793:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829098.900733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829098.900735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829099.220749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829099.732725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829099.732838:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829099.924756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829099.924758:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829099.924767:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829099.924772:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829099.924784:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2106 00000100:00100000:2.0:1787829099.924787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829099.924798:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829099.924809:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829099.924855:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518029568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829099.924882:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829099.924887:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829099.924889:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829099.924894:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2106 00000100:00080000:0.0:1787829099.924899:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829099.924904:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829099.924917:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829099.924933:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518029696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829099.924946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829099.924949:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518029568:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829099.924950:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829099.924953:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829099.924966:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2106 00000100:00080000:0.0:1787829099.924972:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829099.924979:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829099.924983:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829099.924999:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518029824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829099.925011:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829099.925014:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829099.925015:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829099.925018:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2106 00000100:00080000:0.0:1787829099.925021:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829099.925023:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829099.925026:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829099.925033:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518029952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829099.925040:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829099.925043:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829099.925044:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829099.925051:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2106 00000100:00080000:0.0:1787829099.925055:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829099.925055:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829099.925064:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829099.925070:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829099.925077:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518030080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829099.925086:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829099.925089:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829099.925090:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829099.925095:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2106 00000100:00080000:0.0:1787829099.925098:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829099.925101:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829099.925106:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829099.925119:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518030208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829099.925128:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829099] req@ffff8b2e88042580 x1874672518029568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829099.925138:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829099.925151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829099.925154:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829099.925165:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2106 00000100:00080000:0.0:1787829099.925168:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829099.925174:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829099.925181:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518029568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829099.925183:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829099.925196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518030336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829099.925201:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829099.925210:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829099.925217:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518029568:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829099.925240:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518029696:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829099.925262:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829099.925272:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829099] req@ffff8b2e88041e00 x1874672518029696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829099.925281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518029696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829099.925288:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829099.925290:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829099.925292:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518029696:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829099.925297:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518029824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829099.925309:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829099.925315:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829099] req@ffff8b2e88040b40 x1874672518029824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829099.925323:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518029824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829099.925332:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829099.925334:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829099.925337:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518029824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829099.925340:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518029952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829099.925350:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829099.925357:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829099] req@ffff8b2e880412c0 x1874672518029952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829099.925382:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518029952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829099.925392:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829099.925396:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829099.925400:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518029952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829099.925407:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518030080:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829099.925428:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829099.925440:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829099] req@ffff8b2e880421c0 x1874672518030080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829099.925448:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518030080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829099.925464:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829099.925466:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829099.925469:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518030080:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829099.925474:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518030208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829099.925492:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829099.925504:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829099] req@ffff8b2e88042d00 x1874672518030208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829099.925516:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518030208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829099.925525:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829099.925528:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829099.925532:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518030208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829099.925539:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518030336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829099.925556:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829099.925565:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829099/real 1787829099] req@ffff8b2e88040780 x1874672518030336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829099.925573:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518030336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829124 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829099.925579:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829099.925581:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829099.925583:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518030336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829100.244765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829100.756727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829100.756773:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829100.948700:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829100.948712:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829101.268732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829101.780756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829101.780832:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829101.972785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829101.972786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829102.292800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829102.804736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829102.804815:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829102.996759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829102.996784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829103.316738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829103.828758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829103.828797:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829104.020735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829104.020743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829104.340771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829104.852719:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829104.852770:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829105.044762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829105.044767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829105.044777:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829105.044784:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829105.044789:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829105.044799:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2111 00000100:00080000:0.0:1787829105.044811:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829105.044820:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829105.044864:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518030464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829105.044900:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829105.044904:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829105.044906:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829105.044910:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2111 00000100:00080000:0.0:1787829105.044914:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829105.044918:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829105.044928:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829105.044931:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518030464:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829105.044945:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518030592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829105.044954:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829105.044957:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829105.044959:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829105.044962:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2111 00000100:00080000:0.0:1787829105.044964:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829105.044967:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829105.044970:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829105.044979:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518030720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829105.044986:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829105.044988:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829105.044989:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829105.044992:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2111 00000100:00080000:0.0:1787829105.044995:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829105.044998:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829105.045002:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829105.045004:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829105.045013:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518030848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829105.045019:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829105.045022:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829105.045023:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829105.045027:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2111 00000100:00080000:0.0:1787829105.045029:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829105.045032:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829105.045034:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829105.045041:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518030976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829105.045047:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829105.045050:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829105.045052:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829105.045055:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829105/real 1787829105] req@ffff8b2e88043840 x1874672518030464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829105.045057:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2111 00000100:00080000:0.0:1787829105.045063:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829105.045067:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829105.045074:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829105.045088:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518031104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829105.045094:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518030464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829105.045119:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829105.045123:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829105.045124:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829105.045131:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518030464:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829105.045132:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829105.045137:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829105.045144:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2111 00000100:00080000:0.0:1787829105.045150:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829105.045155:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829105.045159:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518030592:192.168.203.101@tcp:38:kworker.0 00000008:00100000:0.0:1787829105.045161:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829105.045178:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518031232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829105.045187:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829105.045201:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829105/real 1787829105] req@ffff8b2e88040f00 x1874672518030592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829105.045211:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518030592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829105.045217:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829105.045219:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829105.045221:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518030592:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829105.045226:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518030720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829105.045239:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829105.045246:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829105/real 1787829105] req@ffff8b2e88041a40 x1874672518030720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829105.045254:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518030720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829105.045259:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829105.045262:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829105.045264:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518030720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829105.045267:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518030848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829105.045277:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829105.045283:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829105/real 1787829105] req@ffff8b2e88040000 x1874672518030848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829105.045290:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518030848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829105.045295:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829105.045297:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829105.045299:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518030848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829105.045303:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518030976:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829105.045313:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829105.045319:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829105/real 1787829105] req@ffff8b2e880430c0 x1874672518030976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829105.045327:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518030976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829105.045346:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829105.045348:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829105.045350:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518030976:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829105.045355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518031104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829105.045370:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829105.045377:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829105/real 1787829105] req@ffff8b2e88043480 x1874672518031104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829105.045385:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518031104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829105.045391:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829105.045393:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829105.045395:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518031104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829105.045407:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518031232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829105.045417:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829105.045423:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829105/real 1787829105] req@ffff8b2e88042940 x1874672518031232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829105.045431:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518031232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829130 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829105.045441:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829105.045443:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829105.045445:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518031232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829105.364722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829105.876751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829105.876819:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829106.068738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829106.068738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829106.388765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829106.900725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829106.900776:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829107.092767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829107.092777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829107.412775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829107.924715:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829107.924808:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829108.116767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829108.116772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829108.436755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829108.948793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829108.948838:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829109.140720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829109.140740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829109.460738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829109.972730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829109.972779:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829110.164734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829110.164744:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829110.164745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829110.164753:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829110.164757:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829110.164767:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2116 00000100:00080000:1.0:1787829110.164782:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829110.164792:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829110.164840:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518031360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829110.164869:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829110.164873:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829110.164875:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829110.164880:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2116 00000100:00080000:1.0:1787829110.164885:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829110.164890:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829110.164901:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829110.164910:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518031360:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829110.164915:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518031488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829110.164925:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829110.164928:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829110.164930:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829110.164933:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2116 00000100:00080000:1.0:1787829110.164936:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829110.164939:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829110.164942:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829110.164952:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518031616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829110.164960:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829110.164963:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829110.164964:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829110.164966:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2116 00000100:00080000:1.0:1787829110.164969:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829110.164971:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829110.164975:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829110.164984:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518031744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829110.164991:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829110.164995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829110.164999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829110.165001:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829110.165005:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2116 00000100:00080000:1.0:1787829110.165007:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829110.165010:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829110.165013:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829110.165020:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518031872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829110.165027:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829110.165029:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829110.165030:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829110.165034:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2116 00000100:00080000:1.0:1787829110.165039:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829110.165039:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829110/real 1787829110] req@ffff8b2e88077840 x1874672518031360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829110.165046:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829110.165053:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829110.165064:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518032000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829110.165066:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518031360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829110.165085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829110.165089:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829110.165089:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829110.165091:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829110.165095:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829110.165098:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2116 00000100:00080000:1.0:1787829110.165102:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829110.165103:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518031360:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829110.165108:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829110.165112:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829110.165119:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518032128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829110.165131:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518031488:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829110.165150:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829110.165159:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829110/real 1787829110] req@ffff8b2e88074780 x1874672518031488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829110.165169:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518031488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829110.165175:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829110.165178:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829110.165180:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518031488:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829110.165184:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518031616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829110.165196:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829110.165202:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829110/real 1787829110] req@ffff8b2e88076940 x1874672518031616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829110.165209:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518031616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829110.165215:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829110.165216:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829110.165218:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518031616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829110.165222:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518031744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829110.165231:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829110.165237:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829110/real 1787829110] req@ffff8b2e88074f00 x1874672518031744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829110.165244:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518031744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829110.165254:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829110.165257:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829110.165259:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518031744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829110.165262:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518031872:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829110.165271:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829110.165277:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829110/real 1787829110] req@ffff8b2e88075680 x1874672518031872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829110.165284:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518031872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829110.165290:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829110.165292:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829110.165294:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518031872:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829110.165310:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518032000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829110.165326:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829110.165333:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829110/real 1787829110] req@ffff8b2e88075a40 x1874672518032000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829110.165341:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518032000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829110.165346:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829110.165348:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829110.165351:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518032000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829110.165355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518032128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829110.165366:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829110.165372:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829110/real 1787829110] req@ffff8b2e88077480 x1874672518032128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829110.165379:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518032128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829135 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829110.165385:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829110.165387:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829110.165389:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518032128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829110.484778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829110.996736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829110.996820:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829111.188738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829111.188764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829111.508715:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829112.020761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829112.020837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829112.212755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829112.212755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829112.532726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829113.044747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829113.044813:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829113.236727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829113.236789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829113.556729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829114.068723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829114.068800:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829114.260723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829114.260748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829114.580784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829115.092750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829115.092833:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829115.284707:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829115.284765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829115.284777:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.284785:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829115.284790:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.284800:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2121 00000100:00080000:2.0:1787829115.284812:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.284820:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829115.284861:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518032256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829115.284884:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.284887:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829115.284888:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.284892:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2121 00000100:00080000:2.0:1787829115.284895:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.284898:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829115.284907:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829115.284918:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518032256:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829115.284919:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518032384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829115.284932:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.284934:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829115.284936:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.284939:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2121 00000100:00080000:2.0:1787829115.284942:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.284943:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829115.284947:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829115.284955:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518032512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829115.284962:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.284964:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829115.284965:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.284969:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2121 00000100:00080000:2.0:1787829115.284971:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.284974:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829115.284976:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829115.284982:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518032640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829115.284989:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829115.284995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.284998:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829115.285000:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.285004:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2121 00000100:00080000:2.0:1787829115.285006:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.285008:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829115.285011:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829115.285017:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518032768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829115.285023:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.285026:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829115.285027:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.285031:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2121 00000100:00080000:2.0:1787829115.285033:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.285035:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829115.285038:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829115.285041:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff8b2e8801e940 x1874672518032256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829115.285048:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518032896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829115.285058:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829115.285059:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829115.285060:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829115.285063:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2121 00000100:00080000:2.0:1787829115.285066:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829115.285069:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829115.285074:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829115.285074:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518032256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829115.285084:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518033024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829115.285096:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.285104:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829115.285110:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518032256:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829115.285131:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518032384:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829115.285152:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829115.285162:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff8b2e8801e580 x1874672518032384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829115.285171:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518032384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.285177:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.285180:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829115.285182:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518032384:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829115.285186:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518032512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829115.285197:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829115.285204:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff8b2e8801c000 x1874672518032512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829115.285211:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518032512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.285217:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.285218:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829115.285220:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518032512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829115.285224:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518032640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829115.285234:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829115.285240:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff8b2e8801cf00 x1874672518032640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829115.285247:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518032640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.285252:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.285254:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829115.285256:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518032640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829115.285259:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518032768:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829115.285288:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829115.285295:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff8b2e8801f480 x1874672518032768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829115.285303:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518032768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.285309:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.285312:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829115.285314:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518032768:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829115.285317:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518032896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829115.285327:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829115.285333:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff8b2e8801fc00 x1874672518032896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829115.285345:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518032896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.285350:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.285352:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829115.285354:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518032896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829115.285357:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518033024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829115.285366:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829115.285372:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829115/real 1787829115] req@ffff8b2e8801da40 x1874672518033024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829115.285381:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518033024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829140 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829115.285389:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829115.285392:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829115.285395:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518033024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829115.604757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829116.116752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829116.116788:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829116.308727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829116.308727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829116.628763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829117.140740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829117.140801:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829117.332746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829117.332746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829117.652767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829118.164752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829118.164855:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829118.356717:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829118.356733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829118.676744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829119.188782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829119.188832:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829119.380730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829119.380730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829119.700780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829120.212791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829120.212872:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829120.404744:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829120.404750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829120.404760:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829120.404765:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829120.404775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829120.404776:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2126 00000100:00080000:2.0:1787829120.404787:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829120.404795:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829120.404834:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518033152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829120.404854:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829120.404857:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829120.404858:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829120.404862:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2126 00000100:00080000:2.0:1787829120.404865:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829120.404868:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829120.404876:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829120.404886:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518033280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829120.404891:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518033152:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829120.404898:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829120.404902:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829120.404903:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829120.404905:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2126 00000100:00080000:2.0:1787829120.404907:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829120.404909:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829120.404913:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829120.404922:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518033408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829120.404928:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829120.404930:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829120.404932:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829120.404934:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2126 00000100:00080000:2.0:1787829120.404951:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829120.404956:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829120.404958:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829120.404963:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829120.404973:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518033536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829120.404980:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829120.404982:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829120.404984:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829120.404987:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2126 00000100:00080000:2.0:1787829120.404990:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829120.404993:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829120.404997:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829120.404998:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff8b2e8801c3c0 x1874672518033152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829120.405007:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518033664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829120.405016:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829120.405020:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829120.405020:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518033152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829120.405024:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829120.405032:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2126 00000100:00080000:2.0:1787829120.405037:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829120.405038:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829120.405040:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829120.405045:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829120.405045:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829120.405050:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518033152:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829120.405056:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518033792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829120.405063:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829120.405066:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829120.405067:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829120.405071:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2126 00000100:00100000:3.0:1787829120.405074:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518033280:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829120.405075:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829120.405082:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829120.405086:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829120.405095:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518033920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829120.405102:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829120.405117:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff8b2e8801cb40 x1874672518033280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829120.405128:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518033280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829120.405134:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829120.405136:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829120.405139:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518033280:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829120.405143:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518033408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829120.405155:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829120.405162:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff8b2e8801ed00 x1874672518033408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829120.405169:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518033408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829120.405179:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829120.405181:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829120.405183:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518033408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829120.405186:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518033536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829120.405195:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829120.405201:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff8b2e8801c780 x1874672518033536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829120.405208:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518033536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829120.405214:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829120.405216:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829120.405218:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518033536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829120.405221:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518033664:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829120.405252:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829120.405261:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff8b2e8801f0c0 x1874672518033664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829120.405269:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518033664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829120.405275:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829120.405277:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829120.405280:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518033664:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829120.405284:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518033792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829120.405294:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829120.405300:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff8b2e8801d680 x1874672518033792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829120.405308:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518033792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829120.405313:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829120.405315:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829120.405316:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518033792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829120.405328:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518033920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829120.405339:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829120.405345:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829120/real 1787829120] req@ffff8b2e8801de00 x1874672518033920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829120.405353:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518033920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829145 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829120.405358:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829120.405359:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829120.405361:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518033920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829120.724748:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829121.236751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829121.236811:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829121.428743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829121.428746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829121.748710:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829122.260719:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829122.260808:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829122.452747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829122.452747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829122.772705:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829123.284762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829123.284819:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829123.476748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829123.476772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829123.796747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829124.308726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829124.308816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829124.500757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829124.500757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829124.820728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829125.332803:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829125.332834:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829125.524778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829125.524781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829125.524809:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.524823:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829125.524827:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.524836:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2131 00000100:00080000:2.0:1787829125.524845:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.524851:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829125.524892:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518034048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829125.524929:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.524933:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829125.524936:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.524940:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2131 00000100:00080000:2.0:1787829125.524945:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.524949:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829125.524958:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829125.524970:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518034176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829125.524980:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518034048:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829125.524982:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.524988:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829125.524993:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.524997:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2131 00000100:00080000:2.0:1787829125.524999:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.525001:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829125.525004:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829125.525013:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518034304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829125.525020:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.525022:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829125.525023:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.525026:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2131 00000100:00080000:2.0:1787829125.525040:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.525043:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829125.525046:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829125.525053:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518034432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829125.525059:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.525062:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829125.525063:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.525066:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2131 00000100:00080000:2.0:1787829125.525068:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.525070:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829125.525073:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829125.525085:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518034560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829125.525092:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.525095:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829125.525096:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.525099:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2131 00000100:00080000:2.0:1787829125.525100:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.525103:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829125.525106:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829125.525109:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518034176:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787829125.525117:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518034688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829125.525129:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829125.525131:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829125.525132:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829125.525135:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2131 00000100:00080000:2.0:1787829125.525142:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829125.525145:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829125.525149:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829125.525157:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518034304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787829125.525159:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518034816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829125.525213:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518034432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829125.525243:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518034560:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829125.525278:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518034688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829125.525299:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518034816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829125.844758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829126.356719:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829126.356822:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829126.548775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829126.548866:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829126.868758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829127.380758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829127.380830:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829127.572745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829127.572822:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829127.892767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829128.404793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829128.404805:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829128.596777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829128.596942:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:3.0:1787829128.597019:0:2336:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:3.0:1787829128.597067:0:2336:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829128.597083:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829128.597124:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829128.597134:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829128.597145:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829128.597163:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787829128.597183:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829128] req@ffff8b2e8801d680 x1874672518034176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787829128.597191:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829128.597205:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787829128.597220:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518034176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829128.597233:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829128.597237:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829128.597242:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518034176:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829128.597250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829128] req@ffff8b2e8801f0c0 x1874672518034304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829128.597263:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518034304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829128.597270:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829128.597274:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829128.597277:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518034304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829128.597284:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829128] req@ffff8b2e8801c780 x1874672518034432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829128.597295:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518034432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829128.597302:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829128.597306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829128.597309:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518034432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829128.597315:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829128] req@ffff8b2e8801ed00 x1874672518034560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829128.597326:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518034560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829128.597334:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829128.597338:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829128.597342:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518034560:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829128.597348:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829128] req@ffff8b2e8801de00 x1874672518034048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829128.597360:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518034048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829128.597368:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829128.597371:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829128.597374:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518034048:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829128.597400:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829128] req@ffff8b2e8801cb40 x1874672518034688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829128.597412:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518034688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829128.597420:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829128.597423:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829128.597427:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518034688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829128.597434:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829125/real 1787829128] req@ffff8b2e8801c3c0 x1874672518034816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829128.597446:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518034816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829150 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829128.597458:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829128.597462:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829128.597465:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518034816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829128.916740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829129.428746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829129.428825:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829129.620780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829129.620862:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829129.940775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829130.452773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829130.452862:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829130.644822:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829130.644823:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829130.644829:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829130.644839:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829130.644848:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829130.644864:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2137 00000100:00080000:0.0:1787829130.644880:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829130.644892:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829130.644944:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518034944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829130.644975:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829130.644978:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829130.644979:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829130.644983:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2137 00000100:00080000:0.0:1787829130.644987:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829130.644992:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829130.645005:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829130.645020:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518034944:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829130.645025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518035072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829130.645039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829130.645043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829130.645046:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829130.645050:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2137 00000100:00080000:0.0:1787829130.645054:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829130.645057:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829130.645062:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829130.645073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518035200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829130.645080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829130.645082:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829130.645084:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829130.645086:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2137 00000100:00080000:0.0:1787829130.645091:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829130.645093:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829130.645096:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829130.645104:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518035328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829130.645111:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829130.645114:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829130.645118:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829130.645120:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829130.645125:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2137 00000100:00080000:0.0:1787829130.645128:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829130.645132:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829130.645149:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829130.645180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518035456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829130.645191:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff8b2e880403c0 x1874672518034944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829130.645193:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829130.645202:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829130.645206:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829130.645220:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2137 00000100:00080000:0.0:1787829130.645225:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829130.645227:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518034944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829130.645233:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829130.645244:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829130.645250:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829130.645258:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829130.645262:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518035584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829130.645267:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518034944:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829130.645284:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829130.645288:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829130.645290:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829130.645295:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2137 00000100:00080000:0.0:1787829130.645300:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829130.645307:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829130.645308:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518035072:192.168.203.101@tcp:38:kworker.0 00000008:00100000:0.0:1787829130.645314:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829130.645332:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518035712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829130.645346:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829130.645366:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff8b2e88042940 x1874672518035072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829130.645383:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518035072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829130.645394:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829130.645398:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829130.645402:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518035072:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829130.645409:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518035200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829130.645434:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829130.645448:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff8b2e88043480 x1874672518035200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829130.645461:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518035200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829130.645476:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829130.645480:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829130.645483:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518035200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829130.645490:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518035328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829130.645512:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829130.645524:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff8b2e880430c0 x1874672518035328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829130.645538:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518035328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829130.645994:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829130.645999:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829130.646002:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518035328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829130.646009:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518035456:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829130.646057:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829130.646074:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff8b2e88040000 x1874672518035456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829130.646090:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518035456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829130.646101:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829130.646104:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829130.646108:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518035456:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829130.646113:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518035584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829130.646132:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829130.646140:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff8b2e88041a40 x1874672518035584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829130.646149:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518035584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829130.646167:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829130.646171:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829130.646175:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518035584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829130.646223:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518035712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829130.646250:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829130.646290:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829130/real 1787829130] req@ffff8b2e88040f00 x1874672518035712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829130.646300:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518035712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829155 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829130.646306:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829130.646309:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829130.646311:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518035712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829130.964738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829131.476737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829131.476813:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829131.668743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829131.668841:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829131.988756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829132.500813:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829132.500868:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829132.692741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829132.692741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829133.012729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829133.524734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829133.524794:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829133.716739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829133.716762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829134.036711:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829134.548760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829134.548812:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829134.740763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829134.740874:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829135.060703:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829135.572744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829135.572817:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829135.764741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829135.764742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829135.764746:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829135.764755:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829135.764760:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829135.764769:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2142 00000100:00080000:1.0:1787829135.764781:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829135.764788:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829135.764823:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518035840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829135.764844:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829135.764847:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829135.764848:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829135.764851:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2142 00000100:00080000:1.0:1787829135.764854:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829135.764856:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829135.764866:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829135.764876:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518035968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829135.764883:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518035840:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829135.764887:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829135.764893:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829135.764894:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829135.764902:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2142 00000100:00080000:1.0:1787829135.764904:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829135.764906:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829135.764910:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829135.764917:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518036096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829135.764923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829135.764926:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829135.764927:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829135.764929:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2142 00000100:00080000:1.0:1787829135.764931:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829135.764933:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829135.764936:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829135.764943:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518036224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829135.764949:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829135.764952:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829135.764954:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829135.764957:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2142 00000100:00080000:1.0:1787829135.764960:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829135.764962:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829135.764966:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829135.764968:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829135.764975:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518036352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829135.764982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829135.764984:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829135.764986:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829135.764989:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2142 00000100:00080000:1.0:1787829135.764993:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829135.764995:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829135.764998:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829135.765004:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518036480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829135.765010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829135.765013:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829135.765014:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829135.765017:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2142 00000100:00080000:1.0:1787829135.765021:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829135.765022:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff8b2e88077c00 x1874672518035840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829135.765026:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829135.765032:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829135.765044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518036608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829135.765055:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518035840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829135.765069:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829135.765073:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829135.765078:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518035840:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829135.765101:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518035968:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829135.765145:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829135.765161:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff8b2e88077480 x1874672518035968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829135.765176:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518035968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829135.765186:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829135.765190:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829135.765194:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518035968:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829135.765201:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518036096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829135.765223:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829135.765235:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff8b2e88075a40 x1874672518036096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829135.765249:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518036096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829135.765258:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829135.765261:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829135.765265:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518036096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829135.765271:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518036224:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829135.765291:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829135.765302:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff8b2e88075680 x1874672518036224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829135.765315:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518036224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829135.765324:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829135.765327:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829135.765331:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518036224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829135.765336:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518036352:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829135.765356:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829135.765368:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff8b2e88074f00 x1874672518036352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829135.765381:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518036352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829135.765390:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829135.765394:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829135.765397:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518036352:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829135.765403:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518036480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829135.765425:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829135.765436:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff8b2e88076940 x1874672518036480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829135.765449:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518036480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829135.765457:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829135.765461:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829135.765464:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518036480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829135.765471:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518036608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829135.765490:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829135.765502:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829135/real 1787829135] req@ffff8b2e88074780 x1874672518036608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829135.765514:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518036608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829160 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829135.765524:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829135.765527:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829135.765531:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518036608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829136.084723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829136.596752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829136.596862:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829136.788749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829136.788752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829137.108715:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829137.620713:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829137.620749:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829137.812722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829137.812722:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829138.132740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829138.644752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829138.644808:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829138.836762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829138.836767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829139.156795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829139.668745:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829139.668816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829139.860731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829139.860732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829140.180737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829140.692764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829140.692794:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829140.884739:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829140.884751:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829140.884757:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829140.884771:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2147 00000100:00080000:1.0:1787829140.884787:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829140.884789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829140.884789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829140.884800:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829140.884853:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518036736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829140.884877:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829140.884880:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829140.884882:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829140.884885:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2147 00000100:00080000:1.0:1787829140.884888:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829140.884891:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829140.884901:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829140.884913:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518036864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829140.884921:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518036736:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829140.884930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829140.884934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829140.884936:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829140.884939:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2147 00000100:00080000:1.0:1787829140.884941:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829140.884944:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829140.884950:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829140.884960:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518036992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829140.884966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829140.884969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829140.884970:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829140.884973:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2147 00000100:00080000:1.0:1787829140.884976:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829140.884979:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829140.884983:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829140.884992:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829140.884993:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518037120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829140.885003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829140.885007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829140.885009:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829140.885015:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2147 00000100:00080000:1.0:1787829140.885022:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829140.885026:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829140.885031:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829140.885043:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518037248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829140.885049:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff8b2e88077840 x1874672518036736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829140.885072:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829140.885099:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829140.885103:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829140.885108:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2147 00000100:00080000:1.0:1787829140.885113:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829140.885115:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518036736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829140.885119:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829140.885131:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829140.885138:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829140.885146:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829140.885147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518037376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829140.885157:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518036736:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829140.885170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829140.885173:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829140.885175:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829140.885179:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2147 00000100:00080000:1.0:1787829140.885182:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829140.885184:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829140.885189:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829140.885192:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518036864:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829140.885201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518037504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829140.885227:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829140.885239:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff8b2e88076580 x1874672518036864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829140.885248:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518036864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829140.885255:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829140.885257:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829140.885260:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518036864:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829140.885264:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518036992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829140.885277:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829140.885283:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff8b2e88076d00 x1874672518036992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829140.885291:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518036992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829140.885296:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829140.885305:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829140.885308:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518036992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829140.885312:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518037120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829140.885323:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829140.885330:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff8b2e880770c0 x1874672518037120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829140.885338:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518037120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829140.885343:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829140.885346:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829140.885348:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518037120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829140.885352:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518037248:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829140.885361:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829140.885367:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff8b2e88074000 x1874672518037248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829140.885374:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518037248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829140.885383:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829140.885386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829140.885389:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518037248:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829140.885403:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518037376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829140.885425:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829140.885438:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff8b2e880761c0 x1874672518037376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829140.885450:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518037376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829140.885456:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829140.885458:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829140.885460:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518037376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829140.885476:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518037504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829140.885491:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829140.885498:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829140/real 1787829140] req@ffff8b2e88075e00 x1874672518037504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829140.885505:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518037504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829165 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829140.885511:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829140.885513:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829140.885514:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518037504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829141.204783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829141.716734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829141.716851:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829141.908779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829141.908785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829142.228738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829142.740744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829142.740821:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829142.932739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829142.932739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829143.252774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829143.764723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829143.764776:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829143.956759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829143.956760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829144.276724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829144.788712:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829144.788779:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829144.980712:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829144.980725:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829145.300718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829145.812703:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829145.812794:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829146.004758:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829146.004758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829146.004759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829146.004770:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829146.004776:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829146.004787:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2152 00000100:00080000:1.0:1787829146.004809:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829146.004818:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829146.004857:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518037632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829146.004884:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829146.004888:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829146.004890:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829146.004894:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2152 00000100:00080000:1.0:1787829146.004898:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829146.004903:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829146.004913:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829146.004913:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518037632:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829146.004944:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518037760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829146.004956:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829146.004960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829146.004962:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829146.004965:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2152 00000100:00080000:1.0:1787829146.004970:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829146.004975:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829146.004979:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829146.004987:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518037888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829146.004992:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829146.005000:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829146.005003:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829146.005004:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829146.005007:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2152 00000100:00080000:1.0:1787829146.005010:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829146.005013:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829146.005016:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829146.005023:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518038016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829146.005037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829146.005039:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829146.005041:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829146.005044:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2152 00000100:00080000:1.0:1787829146.005046:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829146.005063:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829146.005065:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829146/real 1787829146] req@ffff8b2e88074b40 x1874672518037632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829146.005068:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829146.005083:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518038144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829146.005090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829146.005095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829146.005095:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518037632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829146.005099:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829146.005105:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2152 00000100:00080000:1.0:1787829146.005111:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829146.005111:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829146.005115:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829146.005118:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787829146.005120:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829146.005123:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518037632:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829146.005132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518038272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829146.005138:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829146.005141:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829146.005143:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829146.005146:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518037760:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829146.005149:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2152 00000100:00080000:1.0:1787829146.005153:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829146.005155:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829146.005158:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829146.005168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518038400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829146.005175:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829146.005190:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829146/real 1787829146] req@ffff8b2e88075e00 x1874672518037760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829146.005199:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518037760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829146.005207:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829146.005211:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829146.005215:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518037760:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829146.005221:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518037888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829146.005256:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829146.005270:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829146/real 1787829146] req@ffff8b2e880761c0 x1874672518037888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829146.005283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518037888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829146.005299:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829146.005303:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829146.005307:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518037888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829146.005313:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518038016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829146.005329:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829146.005337:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829146/real 1787829146] req@ffff8b2e88074000 x1874672518038016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829146.005345:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518038016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829146.005351:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829146.005365:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829146.005367:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518038016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829146.005370:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518038144:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829146.005380:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829146.005386:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829146/real 1787829146] req@ffff8b2e880770c0 x1874672518038144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829146.005393:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518038144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829146.005399:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829146.005400:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829146.005403:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518038144:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829146.005406:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518038272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829146.005416:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829146.005431:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829146/real 1787829146] req@ffff8b2e88076d00 x1874672518038272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829146.005438:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518038272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829146.005443:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829146.005445:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829146.005447:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518038272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829146.005464:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518038400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829146.005483:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829146.005495:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829146/real 1787829146] req@ffff8b2e88076580 x1874672518038400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829146.005507:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518038400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829171 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829146.005513:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829146.005515:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829146.005517:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518038400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829146.324740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829146.836709:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829146.836744:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829147.028716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829147.028716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829147.348768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829147.860708:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829147.860788:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829148.052759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829148.052759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829148.372719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829148.884746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829148.884863:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829149.076772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829149.076772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829149.396763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829149.908755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829149.908780:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829150.100780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829150.100780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829150.420781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829150.932723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829150.932833:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829151.124732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829151.124738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829151.124740:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829151.124746:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829151.124770:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829151.124781:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2157 00000100:00080000:1.0:1787829151.124791:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829151.124796:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829151.124842:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518038528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829151.124863:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829151.124865:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829151.124867:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829151.124870:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2157 00000100:00080000:1.0:1787829151.124872:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829151.124875:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829151.124883:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829151.124892:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518038528:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829151.124894:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518038656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829151.124906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829151.124908:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829151.124909:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829151.124912:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2157 00000100:00080000:1.0:1787829151.124915:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829151.124917:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829151.124920:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829151.124927:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518038784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829151.124949:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829151.124952:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829151.124953:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829151.124955:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2157 00000100:00080000:1.0:1787829151.124958:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829151.124960:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829151.124965:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787829151.124967:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829151.124976:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518038912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829151.124982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829151.124985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829151.124986:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829151.124988:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2157 00000100:00080000:1.0:1787829151.124990:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829151.124993:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829151.124995:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829151.125003:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518039040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829151.125010:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829151/real 1787829151] req@ffff8b2e88077840 x1874672518038528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829151.125012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829151.125031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829151.125035:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829151.125054:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2157 00000100:00080000:1.0:1787829151.125058:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829151.125062:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518038528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829151.125063:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829151.125070:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829151.125079:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829151.125084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518039168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829151.125085:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829151.125092:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518038528:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829151.125100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829151.125103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829151.125104:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829151.125121:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2157 00000100:00080000:1.0:1787829151.125123:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829151.125126:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829151.125130:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518038656:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787829151.125132:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829151.125144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518039296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829151.125156:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829151.125172:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829151/real 1787829151] req@ffff8b2e88074780 x1874672518038656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829151.125188:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518038656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829151.125208:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829151.125212:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829151.125216:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518038656:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829151.125222:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518038784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829151.125243:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829151.125255:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829151/real 1787829151] req@ffff8b2e88076940 x1874672518038784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829151.125266:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518038784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829151.125274:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829151.125277:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829151.125281:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518038784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829151.125287:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518038912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829151.125303:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829151.125313:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829151/real 1787829151] req@ffff8b2e88074f00 x1874672518038912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829151.125325:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518038912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829151.125333:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829151.125336:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829151.125340:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518038912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829151.125346:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518039040:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829151.125362:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829151.125372:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829151/real 1787829151] req@ffff8b2e88075680 x1874672518039040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829151.125394:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518039040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829151.125403:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829151.125406:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829151.125409:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518039040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829151.125414:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518039168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829151.125429:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829151.125438:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829151/real 1787829151] req@ffff8b2e88075a40 x1874672518039168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829151.125461:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518039168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829151.125469:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829151.125472:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829151.125476:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518039168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829151.125494:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518039296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829151.125510:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829151.125539:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829151/real 1787829151] req@ffff8b2e88077480 x1874672518039296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829151.125551:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518039296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829176 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829151.125559:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829151.125563:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829151.125565:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518039296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829151.444802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829151.956751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829151.956857:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829152.148744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829152.148744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829152.468764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829152.980733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829152.980802:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829153.172789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829153.172789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829153.492805:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829154.004746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829154.004870:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829154.196758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829154.196803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829154.516744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829155.028755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829155.028830:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829155.220705:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829155.220728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829155.540731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829156.052760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829156.052784:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829156.244777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829156.244856:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829156.244917:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.244922:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829156.244926:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.244935:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2162 00000100:00080000:1.0:1787829156.244944:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829156.244950:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829156.245003:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518039424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829156.245038:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.245041:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829156.245042:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.245045:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2162 00000100:00080000:1.0:1787829156.245049:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829156.245053:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829156.245063:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829156.245072:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518039424:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829156.245078:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518039552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829156.245088:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.245091:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829156.245093:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.245097:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2162 00000100:00080000:1.0:1787829156.245100:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829156.245103:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829156.245106:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829156.245117:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518039680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829156.245124:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.245126:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829156.245128:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829156.245132:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829156.245137:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2162 00000100:00080000:1.0:1787829156.245143:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829156.245148:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829156.245154:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829156.245168:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518039808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829156.245173:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff8b2e88077c00 x1874672518039424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829156.245197:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829156.245201:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518039424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829156.245202:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829156.245206:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.245216:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2162 00000100:00080000:0.0:1787829156.245219:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829156.245221:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829156.245235:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829156.245238:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829156.245242:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518039424:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787829156.245255:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829156.245261:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518039552:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829156.245276:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518039936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829156.245282:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829156.245301:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829156.245305:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff8b2e88077480 x1874672518039552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829156.245306:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829156.245310:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.245320:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2162 00000100:00080000:1.0:1787829156.245325:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829156.245326:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518039552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.245335:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.245342:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829156.245343:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829156.245351:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518039552:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787829156.245351:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829156.245364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518039680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829156.245367:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518040064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829156.245377:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829156.245379:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829156.245380:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829156.245384:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2162 00000100:00080000:1.0:1787829156.245393:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829156.245396:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829156.245400:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829156.245405:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829156.245412:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff8b2e88075a40 x1874672518039680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829156.245416:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518040192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829156.245448:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518039680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.245460:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.245463:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829156.245468:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518039680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829156.245474:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518039808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829156.245508:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829156.245517:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff8b2e88075680 x1874672518039808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829156.245525:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518039808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.245530:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.245532:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829156.245534:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518039808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829156.245546:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518039936:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829156.245564:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829156.245574:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff8b2e88074f00 x1874672518039936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829156.245588:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518039936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.245605:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.245608:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829156.245610:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518039936:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829156.245615:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518040064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829156.245630:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829156.245637:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff8b2e88076940 x1874672518040064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829156.245662:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518040064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.245668:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.245672:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829156.245674:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518040064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829156.245679:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518040192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829156.245691:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829156.245698:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829156/real 1787829156] req@ffff8b2e88074780 x1874672518040192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829156.245705:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518040192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829181 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829156.245711:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829156.245712:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829156.245715:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518040192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829156.564741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829157.076731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829157.076836:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829157.268742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829157.268742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829157.588769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829158.100765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829158.100824:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829158.292784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829158.292790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829158.612791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829159.124749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829159.124791:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829159.316752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829159.316752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829159.636725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829160.148771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829160.148824:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829160.340771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829160.340778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829160.660786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829161.172733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829161.172869:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829161.364770:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829161.364774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829161.364782:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829161.364788:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829161.364799:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2167 00000100:00100000:0.0:1787829161.364804:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829161.364810:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829161.364822:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829161.364858:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518040320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829161.364878:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829161.364882:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829161.364884:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829161.364889:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2167 00000100:00080000:1.0:1787829161.364892:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829161.364895:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829161.364903:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518040320:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787829161.364904:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829161.364924:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518040448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829161.364934:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829161.364937:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829161.364939:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829161.364954:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2167 00000100:00080000:1.0:1787829161.364958:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829161.364963:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829161.364969:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829161.364981:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518040576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829161.364988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829161.364990:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829161.364992:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829161.365005:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2167 00000100:00080000:1.0:1787829161.365018:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829161.365021:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829161.365025:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829161.365031:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829161.365036:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518040704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829161.365043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829161.365046:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829161.365047:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829161.365050:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2167 00000100:00080000:1.0:1787829161.365054:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829161.365058:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829161.365063:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829161.365074:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518040832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829161.365082:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829161.365084:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829161.365086:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829161.365090:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2167 00000100:00080000:1.0:1787829161.365092:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829161.365094:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff8b2e88077840 x1874672518040320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829161.365099:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829161.365115:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829161.365134:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518040960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829161.365142:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829161.365147:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518040320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829161.365152:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829161.365159:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829161.365173:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2167 00000100:00080000:1.0:1787829161.365182:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829161.365186:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829161.365187:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829161.365198:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787829161.365199:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829161.365210:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518040320:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829161.365216:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518041088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829161.365259:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518040448:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829161.365284:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829161.365293:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff8b2e88076580 x1874672518040448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829161.365302:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518040448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829161.365309:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829161.365312:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829161.365314:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518040448:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829161.365319:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518040576:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829161.365332:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829161.365337:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff8b2e88076d00 x1874672518040576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829161.365345:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518040576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829161.365351:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829161.365353:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829161.365354:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518040576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829161.365358:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518040704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829161.365369:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829161.365374:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff8b2e880770c0 x1874672518040704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829161.365382:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518040704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829161.365387:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829161.365389:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829161.365391:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518040704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829161.365394:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518040832:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829161.365405:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829161.365411:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff8b2e88074000 x1874672518040832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829161.365419:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518040832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829161.365425:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829161.365426:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829161.365428:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518040832:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829161.365432:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518040960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829161.365441:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829161.365447:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff8b2e880761c0 x1874672518040960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829161.365454:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518040960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829161.365459:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829161.365461:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829161.365462:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518040960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829161.365466:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518041088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829161.365475:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829161.365480:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829161/real 1787829161] req@ffff8b2e88075e00 x1874672518041088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829161.365492:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518041088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829186 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829161.365497:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829161.365499:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829161.365501:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518041088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829161.684720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829162.196743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829162.196778:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829162.388734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829162.388736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829162.708741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829163.220735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829163.220813:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829163.412796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829163.412796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829163.732724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829164.244740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829164.244812:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829164.436722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829164.436748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829164.756725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829165.268717:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829165.268765:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829165.460765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829165.460766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829165.780739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829166.292735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829166.292770:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829166.484734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829166.484734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829166.484808:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829166.484813:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829166.484817:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829166.484825:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2172 00000100:00080000:1.0:1787829166.484834:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829166.484840:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829166.484874:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518041216/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829166.484897:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829166.484900:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829166.484902:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829166.484905:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2172 00000100:00080000:1.0:1787829166.484921:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829166.484927:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829166.484936:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829166.484948:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518041216:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829166.484951:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518041344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829166.484962:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829166.484965:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829166.484967:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829166.484969:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2172 00000100:00080000:1.0:1787829166.484973:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829166.484975:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829166.484979:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829166.484988:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518041472/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829166.484995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829166.484998:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829166.484999:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829166.485001:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2172 00000100:00080000:1.0:1787829166.485004:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829166.485007:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787829166.485011:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787829166.485013:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829166.485023:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518041600/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829166.485030:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829166.485033:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829166.485034:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829166.485037:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2172 00000100:00080000:1.0:1787829166.485039:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829166.485042:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829166.485045:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829166.485053:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829166] req@ffff8b2e88074b40 x1874672518041216/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829166.485056:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518041728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829166.485082:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829166.485084:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829166.485089:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518041216/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829166.485089:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829166.485097:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2172 00000100:00080000:1.0:1787829166.485103:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829166.485107:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829166.485108:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829166.485111:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829166.485115:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829166.485119:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518041216:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829166.485123:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518041856/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829166.485135:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829166.485139:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829166.485142:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518041344:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829166.485144:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829166.485154:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2172 00000100:00080000:1.0:1787829166.485160:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829166.485165:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787829166.485170:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787829166.485175:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829166.485183:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829166] req@ffff8b2e88075e00 x1874672518041344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829166.485192:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518041984/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829166.485198:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518041344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829166.485210:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829166.485213:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829166.485215:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518041344:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829166.485221:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518041472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829166.485237:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829166.485244:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829166] req@ffff8b2e880761c0 x1874672518041472/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829166.485252:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518041472/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829166.485257:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829166.485259:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829166.485261:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518041472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829166.485265:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518041600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829166.485276:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829166.485286:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829166] req@ffff8b2e88074000 x1874672518041600/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829166.485307:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518041600/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829166.485316:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829166.485320:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829166.485323:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518041600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829166.485330:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518041728:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829166.485352:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829166.485365:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829166] req@ffff8b2e880770c0 x1874672518041728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829166.485379:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518041728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829166.485385:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829166.485388:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829166.485390:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518041728:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829166.485394:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518041856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829166.485411:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829166.485423:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829166] req@ffff8b2e88076d00 x1874672518041856/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829166.485436:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518041856/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829166.485452:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829166.485456:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829166.485460:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518041856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829166.485482:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518041984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829166.485505:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829166.485518:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829166/real 1787829166] req@ffff8b2e88076580 x1874672518041984/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829166.485532:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518041984/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829191 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829166.485542:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829166.485546:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829166.485549:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518041984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829166.804830:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829167.316756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829167.316836:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829167.508780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829167.508783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829167.828768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829168.340758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829168.340858:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829168.532784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829168.532883:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829168.852737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829169.364864:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829169.364954:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829169.556805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829169.556805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829169.876736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829170.388766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829170.388877:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829170.580766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829170.580766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829170.900743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829171.412787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829171.412830:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829171.604783:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829171.604788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829171.604788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829171.604800:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829171.604808:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829171.604822:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2178 00000100:00080000:1.0:1787829171.604841:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829171.604852:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829171.604961:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518042112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829171.605013:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829171.605018:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829171.605021:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829171.605028:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2178 00000100:00080000:1.0:1787829171.605033:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829171.605039:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829171.605055:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829171.605065:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518042112:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829171.605072:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518042240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829171.605088:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829171.605093:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829171.605095:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829171.605100:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2178 00000100:00080000:1.0:1787829171.605104:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829171.605109:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829171.605115:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829171.605139:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518042368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829171.605166:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829171.605171:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829171.605174:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829171.605175:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829171.605178:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2178 00000100:00080000:1.0:1787829171.605181:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829171.605196:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829171.605201:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829171.605230:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518042496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829171.605244:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829171.605249:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829171.605252:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829171.605259:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2178 00000100:00100000:0.0:1787829171.605263:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff8b2e88077840 x1874672518042112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829171.605265:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829171.605272:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829171.605279:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829171.605293:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518042624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829171.605321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518042112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829171.605324:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829171.605331:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829171.605334:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829171.605341:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829171.605341:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2178 00000100:00080000:1.0:1787829171.605346:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829171.605349:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829171.605353:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829171.605357:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518042112:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787829171.605360:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829171.605390:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518042752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829171.605411:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829171.605416:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829171.605418:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518042240:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829171.605419:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829171.605424:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2178 00000100:00080000:1.0:1787829171.605428:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829171.605433:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829171.605453:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829171.605459:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829171.605463:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518042880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829171.605471:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff8b2e88074780 x1874672518042240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829171.605485:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518042240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.605495:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.605498:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829171.605502:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518042240:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829171.605509:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518042368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829171.605530:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829171.605539:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff8b2e88076940 x1874672518042368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829171.605547:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518042368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.605557:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.605560:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829171.605562:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518042368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829171.605565:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518042496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829171.605577:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829171.605582:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff8b2e88074f00 x1874672518042496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829171.605590:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518042496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.605595:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.605597:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829171.605599:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518042496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829171.605602:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518042624:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829171.605612:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829171.605618:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff8b2e88075680 x1874672518042624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829171.605626:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518042624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.605631:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.605633:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829171.605636:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518042624:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829171.605640:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518042752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829171.605650:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829171.605655:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff8b2e88075a40 x1874672518042752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829171.605662:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518042752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.605668:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.605669:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829171.605671:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518042752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829171.605684:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518042880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829171.605694:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829171.605700:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829171/real 1787829171] req@ffff8b2e88077480 x1874672518042880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829171.605707:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518042880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829196 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829171.605712:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829171.605714:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829171.605717:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518042880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829171.924907:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829172.436727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829172.436818:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829172.628736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829172.628757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829172.948733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829173.460749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829173.460837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829173.652760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829173.652760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829173.972746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829174.484738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829174.484783:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829174.676794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829174.676794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829174.996792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829175.508774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829175.508813:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829175.700765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829175.700812:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829176.020728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829176.532777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829176.532902:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829176.724746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829176.724762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829176.724767:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829176.724777:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829176.724781:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829176.724791:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2183 00000100:00080000:1.0:1787829176.724803:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829176.724812:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829176.724863:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518043008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829176.724888:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829176.724892:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829176.724894:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829176.724899:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2183 00000100:00080000:1.0:1787829176.724903:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829176.724909:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829176.724920:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829176.724927:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518043008:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829176.724943:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518043136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829176.724956:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829176.724960:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829176.724961:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829176.724965:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2183 00000100:00080000:1.0:1787829176.724969:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829176.724972:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829176.724978:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829176.724989:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518043264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829176.724995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829176.724998:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829176.724999:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829176.725002:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2183 00000100:00080000:1.0:1787829176.725005:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829176.725011:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829176.725015:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829176.725020:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829176.725031:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518043392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829176.725039:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829176.725041:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829176.725042:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829176.725046:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2183 00000100:00080000:1.0:1787829176.725049:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829176.725053:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829176.725059:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829176.725068:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff8b2e88077c00 x1874672518043008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829176.725069:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518043520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829176.725102:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829176.725105:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829176.725108:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829176.725111:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2183 00000100:00080000:1.0:1787829176.725113:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829176.725119:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518043008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829176.725120:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829176.725128:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829176.725148:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518043648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829176.725155:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829176.725155:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829176.725166:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829176.725170:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829176.725172:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829176.725181:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518043008:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829176.725182:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2183 00000100:00080000:1.0:1787829176.725194:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829176.725200:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829176.725204:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829176.725212:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518043776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829176.725220:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518043136:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829176.725247:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829176.725261:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff8b2e88077480 x1874672518043136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.725274:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518043136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.725285:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.725289:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829176.725293:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518043136:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829176.725300:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518043264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829176.725326:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829176.725339:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff8b2e88075a40 x1874672518043264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.725352:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518043264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.725358:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.725360:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829176.725363:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518043264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829176.725368:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518043392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829176.725384:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829176.725395:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff8b2e88075680 x1874672518043392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.725408:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518043392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.725428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.725433:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829176.725435:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518043392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829176.725440:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518043520:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829176.725458:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829176.725466:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff8b2e88074f00 x1874672518043520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.725489:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518043520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.725496:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.725498:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829176.725500:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518043520:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829176.725504:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518043648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829176.725516:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829176.725523:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff8b2e88076940 x1874672518043648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.725531:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518043648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.725536:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.725539:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829176.725540:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518043648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829176.725557:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518043776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829176.725576:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829176.725588:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829176/real 1787829176] req@ffff8b2e88074780 x1874672518043776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829176.725601:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518043776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829201 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829176.725615:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829176.725618:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829176.725620:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518043776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829177.044741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829177.556768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829177.556824:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829177.748779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829177.748779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829178.068761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829178.580718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829178.580775:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829178.772704:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829178.772720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829179.092722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829179.604744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829179.604861:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829179.796740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829179.796764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829180.116783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829180.628728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829180.628801:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829180.820718:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829180.820760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829181.140740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829181.652776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829181.652875:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829181.844773:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829181.844773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829181.844779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829181.844782:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829181.844792:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829181.844818:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2188 00000100:00080000:1.0:1787829181.844831:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829181.844839:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829181.844883:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518043904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829181.844906:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829181.844909:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829181.844911:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829181.844914:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2188 00000100:00080000:1.0:1787829181.844917:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829181.844920:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829181.844930:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829181.844939:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518044032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829181.844946:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518043904:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829181.844965:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829181.844968:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829181.844970:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829181.844973:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2188 00000100:00080000:1.0:1787829181.844976:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829181.844979:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829181.844984:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829181.844992:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518044160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829181.844999:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829181.845001:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829181.845003:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829181.845006:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2188 00000100:00080000:1.0:1787829181.845029:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829181.845032:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829181.845036:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829181.845045:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518044288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829181.845047:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829181.845059:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829181.845062:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829181.845064:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829181.845068:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2188 00000100:00080000:1.0:1787829181.845071:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829181.845074:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829181.845077:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829181.845085:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518044416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829181.845093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829181.845096:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829181.845098:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829181.845105:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2188 00000100:00080000:1.0:1787829181.845111:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829181.845115:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff8b2e88077840 x1874672518043904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829181.845121:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829181.845140:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829181.845153:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518043904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829181.845161:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518044544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829181.845169:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.845177:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829181.845183:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518043904:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829181.845185:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829181.845193:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829181.845196:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829181.845200:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2188 00000100:00080000:1.0:1787829181.845205:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829181.845210:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829181.845215:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518044032:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787829181.845221:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829181.845242:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829181.845243:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518044672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829181.845255:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff8b2e88076580 x1874672518044032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.845272:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518044032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.845280:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.845284:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829181.845286:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518044032:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829181.845293:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518044160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829181.845307:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829181.845314:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff8b2e88076d00 x1874672518044160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.845322:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518044160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.845327:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.845329:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829181.845331:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518044160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829181.845335:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518044288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829181.845345:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829181.845351:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff8b2e880770c0 x1874672518044288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.845359:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518044288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.845369:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.845372:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829181.845375:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518044288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829181.845382:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518044416:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829181.845403:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829181.845415:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff8b2e88074000 x1874672518044416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.845428:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518044416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.845435:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.845438:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829181.845440:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518044416:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829181.845445:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518044544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829181.845461:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829181.845468:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff8b2e880761c0 x1874672518044544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.845476:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518044544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.845481:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.845483:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829181.845485:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518044544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829181.845500:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518044672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829181.845512:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829181.845523:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829181/real 1787829181] req@ffff8b2e88075e00 x1874672518044672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829181.845535:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518044672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829206 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829181.845544:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829181.845548:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829181.845552:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518044672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829182.164771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829182.676735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829182.676836:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829182.868756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829182.868756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829183.188730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829183.700745:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829183.700782:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829183.892733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829183.892759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829184.212745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829184.724710:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829184.724770:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829184.916815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829184.916892:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829185.236732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829185.748785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829185.748887:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829185.940725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829185.940751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829186.260743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829186.772728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829186.772759:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829186.964729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829186.964731:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829186.964732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829186.964741:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829186.964745:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829186.964754:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2193 00000100:00080000:1.0:1787829186.964775:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829186.964781:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829186.964818:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518044800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829186.964839:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829186.964842:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829186.964843:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829186.964846:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2193 00000100:00080000:1.0:1787829186.964849:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829186.964851:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829186.964859:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829186.964870:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518044928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829186.964878:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829186.964882:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829186.964884:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518044800:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829186.964884:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829186.964891:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2193 00000100:00080000:1.0:1787829186.964895:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829186.964898:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829186.964901:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829186.964907:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518045056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829186.964918:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829186.964921:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829186.964922:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829186.964925:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2193 00000100:00080000:1.0:1787829186.964940:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829186.964943:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829186.964945:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829186.964953:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829186.964954:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518045184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829186.964962:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829186.964965:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829186.964966:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829186.964969:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2193 00000100:00080000:1.0:1787829186.964976:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829186.964978:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829186.964981:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829186.964987:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518045312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829186.964993:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829186.964995:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829186.964997:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829186.965002:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2193 00000100:00100000:0.0:1787829186.965004:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff8b2e88074b40 x1874672518044800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829186.965005:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829186.965009:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829186.965014:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829186.965024:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518045440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829186.965031:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829186.965034:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829186.965036:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518044800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829186.965036:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829186.965043:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2193 00000100:00080000:1.0:1787829186.965048:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829186.965054:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829186.965059:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829186.965065:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829186.965067:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518045568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829186.965076:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829186.965084:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518044800:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829186.965112:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518044928:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829186.965139:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829186.965154:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff8b2e88075e00 x1874672518044928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.965181:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518044928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.965191:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.965195:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829186.965198:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518044928:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829186.965206:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518045056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829186.965239:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829186.965252:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff8b2e880761c0 x1874672518045056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.965265:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518045056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.965275:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.965280:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829186.965283:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518045056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829186.965291:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518045184:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829186.965322:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829186.965335:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff8b2e88074000 x1874672518045184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.965349:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518045184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.965358:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.965362:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829186.965366:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518045184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829186.965372:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518045312:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829186.965393:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829186.965404:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff8b2e880770c0 x1874672518045312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.965418:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518045312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.965428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.965431:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829186.965435:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518045312:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829186.965441:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518045440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829186.965463:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829186.965476:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff8b2e88076d00 x1874672518045440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.965490:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518045440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.965506:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.965510:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829186.965514:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518045440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829186.965521:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518045568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829186.965542:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829186.965553:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829186/real 1787829186] req@ffff8b2e88076580 x1874672518045568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829186.965566:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518045568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829211 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829186.965574:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829186.965577:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829186.965581:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518045568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829187.284730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829187.796728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829187.796785:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829187.988730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829187.988730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829188.308705:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829188.820784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829188.820850:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829189.012716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829189.012735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829189.332768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829189.844725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829189.844782:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829190.036766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829190.036766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829190.356711:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829190.868865:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829190.868917:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829191.060856:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829191.060861:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829191.380787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829191.892783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829191.892965:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829192.084793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829192.084793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829192.084864:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829192.084897:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829192.084905:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829192.084921:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2198 00000100:00080000:0.0:1787829192.084960:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829192.084969:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829192.085013:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518045696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829192.085043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829192.085047:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829192.085048:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829192.085052:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2198 00000100:00080000:0.0:1787829192.085055:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829192.085058:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829192.085068:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829192.085080:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518045824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829192.085101:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518045696:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829192.085114:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829192.085119:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829192.085122:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829192.085127:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2198 00000100:00080000:0.0:1787829192.085132:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829192.085136:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829192.085142:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829192.085154:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518045952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829192.085166:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829192.085170:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829192.085172:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829192.085175:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2198 00000100:00080000:0.0:1787829192.085179:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829192.085182:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829192.085188:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829192.085197:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518046080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829192.085206:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829192.085210:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829192.085212:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829192.085216:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2198 00000100:00080000:0.0:1787829192.085219:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829192.085239:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829192.085242:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829192.085250:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518046208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829192.085256:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829192.085258:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829192.085259:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829192.085262:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2198 00000100:00080000:0.0:1787829192.085266:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829192.085269:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829192.085272:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829192.085282:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518046336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829192.085301:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518045824:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829192.085321:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829192.085325:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829192.085329:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829192.085337:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2198 00000100:00080000:0.0:1787829192.085345:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829192.085351:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829192.085358:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829192.085373:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518046464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829192.085384:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518045952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829192.085412:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518046080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829192.085436:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518046208:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829192.085458:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518046336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829192.085480:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518046464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829192.404797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829192.916761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829192.916840:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829193.108799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829193.108824:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829193.428747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829193.940829:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829193.940901:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829194.132780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829194.132782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829194.452782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829194.964779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829194.964857:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829195.156798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829195.156798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:0.0:1787829195.157113:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:0.0:1787829195.157164:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829195.157181:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829195.157219:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829195.157231:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829195.157240:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829195.157249:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829195.157257:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829195.157267:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829195.157538:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829192/real 1787829195] req@ffff8b2e88040b40 x1874672518046336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829195.157564:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518046336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829195.157577:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829195.157583:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829195.157588:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518046336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829195.157597:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829192/real 1787829195] req@ffff8b2e88041e00 x1874672518046464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829195.157609:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518046464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829195.157623:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829195.157626:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829195.157630:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518046464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829195.157636:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829192/real 1787829195] req@ffff8b2e88040780 x1874672518045824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829195.157672:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518045824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829195.157688:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829195.157693:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829195.157698:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518045824:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829195.157719:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829192/real 1787829195] req@ffff8b2e88042d00 x1874672518045952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829195.157734:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518045952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829195.157743:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829195.157748:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829195.157845:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518045952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829195.157855:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829192/real 1787829195] req@ffff8b2e880421c0 x1874672518046080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829195.157878:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518046080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829195.157888:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829195.157893:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829195.157897:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518046080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829195.157905:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829192/real 1787829195] req@ffff8b2e880412c0 x1874672518046208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829195.157918:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518046208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829195.157928:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829195.157933:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829195.157936:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518046208:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829195.157944:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829192/real 1787829195] req@ffff8b2e88043840 x1874672518045696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829195.157975:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518045696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829217 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829195.157984:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829195.157988:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829195.157991:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518045696:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829195.476941:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829195.988768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829195.988972:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829196.180778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829196.180791:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829196.500770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829197.012770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829197.012853:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829197.204765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829197.204801:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829197.204805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829197.204808:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829197.204815:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829197.204825:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2203 00000100:00080000:0.0:1787829197.204835:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829197.204842:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829197.204877:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518046592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829197.204925:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829197.204931:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829197.204934:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829197.204940:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2203 00000100:00080000:0.0:1787829197.204945:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829197.204952:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829197.204964:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829197.204968:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518046592:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829197.204983:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518046720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829197.204999:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829197.205015:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829197.205017:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829197.205022:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2203 00000100:00080000:0.0:1787829197.205028:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829197.205033:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829197.205039:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829197.205053:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829197.205056:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518046848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829197.205085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829197.205089:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829197.205092:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829197.205096:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2203 00000100:00080000:0.0:1787829197.205100:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829197.205105:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829197.205111:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829197.205121:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829197/real 1787829197] req@ffff8b2e88042580 x1874672518046592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829197.205128:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518046976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829197.205152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829197.205157:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787829197.205159:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518046592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829197.205160:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829197.205172:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2203 00000100:00080000:0.0:1787829197.205181:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829197.205184:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829197.205189:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829197.205195:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:0.0:1787829197.205199:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829197.205210:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518046592:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829197.205218:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518047104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829197.205232:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829197.205236:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829197.205238:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829197.205245:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518046720:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829197.205246:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2203 00000100:00080000:0.0:1787829197.205253:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829197.205259:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829197.205266:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829197.205281:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518047232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829197.205293:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829197.205294:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829197.205301:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829197.205304:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829197.205310:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2203 00000100:00080000:0.0:1787829197.205315:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829197.205320:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829197/real 1787829197] req@ffff8b2e88043840 x1874672518046720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829197.205324:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829197.205334:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829197.205344:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518046720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829197.205352:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518047360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829197.205365:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829197.205374:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829197.205380:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518046720:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829197.205390:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518046848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829197.205418:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829197.205441:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829197/real 1787829197] req@ffff8b2e880412c0 x1874672518046848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829197.205457:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518046848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829197.205466:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829197.205470:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829197.205473:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518046848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829197.205481:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518046976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829197.205513:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829197.205525:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829197/real 1787829197] req@ffff8b2e880421c0 x1874672518046976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829197.205537:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518046976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829197.205547:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829197.205550:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829197.205554:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518046976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829197.205560:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518047104:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829197.205582:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829197.205594:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829197/real 1787829197] req@ffff8b2e88042d00 x1874672518047104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829197.205607:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518047104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829197.205623:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829197.205626:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829197.205630:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518047104:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829197.205676:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518047232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829197.205714:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829197.206068:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829197/real 1787829197] req@ffff8b2e88040780 x1874672518047232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829197.206085:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518047232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829197.206096:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829197.206101:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829197.206105:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518047232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829197.206123:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518047360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829197.206151:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829197.206168:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829197/real 1787829197] req@ffff8b2e88041e00 x1874672518047360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829197.206196:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518047360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829222 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829197.206203:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829197.206206:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829197.206210:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518047360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829197.524814:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829198.036744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829198.036797:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829198.228787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829198.228789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829198.548746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829199.060791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829199.060859:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829199.252766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829199.252766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829199.572721:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829200.084781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829200.084848:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829200.276768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829200.276768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829200.596708:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829201.108733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829201.108811:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829201.300760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829201.300791:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829201.620760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829202.132818:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829202.132862:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829202.324749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829202.324749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829202.324752:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829202.324760:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829202.324764:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829202.324774:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2208 00000100:00080000:0.0:1787829202.324784:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829202.324791:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829202.324825:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518047488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829202.324845:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829202.324848:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829202.324849:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829202.324852:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2208 00000100:00080000:0.0:1787829202.324854:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829202.324857:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829202.324865:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829202.324874:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518047616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829202.324882:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518047488:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829202.324894:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829202.324897:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829202.324898:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829202.324901:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2208 00000100:00080000:0.0:1787829202.324904:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829202.324906:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829202.324909:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829202.324919:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518047744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829202.324926:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829202.324928:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829202.324929:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829202.324932:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2208 00000100:00080000:0.0:1787829202.324935:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829202.324937:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829202.324940:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829202.324944:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829202.324950:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518047872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829202.324957:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829202.324959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829202.324961:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829202.324964:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2208 00000100:00080000:0.0:1787829202.324967:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829202.324969:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829202.324972:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829202.324979:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518048000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829202.324985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829202.324988:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff8b2e88040b40 x1874672518047488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829202.324989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829202.324993:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829202.325000:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2208 00000100:00080000:0.0:1787829202.325004:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829202.325007:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829202.325012:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829202.325012:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518047488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829202.325024:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518048128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829202.325024:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829202.325034:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829202.325041:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518047488:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829202.325042:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829202.325048:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829202.325050:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829202.325064:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2208 00000100:00080000:0.0:1787829202.325066:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829202.325069:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829202.325071:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829202.325076:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518047616:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829202.325081:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518048256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829202.325105:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829202.325120:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff8b2e88040f00 x1874672518047616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829202.325139:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518047616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829202.325148:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829202.325151:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829202.325155:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518047616:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829202.325162:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518047744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829202.325202:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829202.325211:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff8b2e88041a40 x1874672518047744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829202.325219:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518047744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829202.325225:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829202.325227:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829202.325229:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518047744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829202.325233:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518047872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829202.325243:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829202.325249:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff8b2e88040000 x1874672518047872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829202.325261:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518047872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829202.325267:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829202.325270:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829202.325273:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518047872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829202.325279:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518048000:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829202.325298:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829202.325309:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff8b2e880430c0 x1874672518048000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829202.325322:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518048000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829202.325328:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829202.325330:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829202.325332:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518048000:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829202.325337:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518048128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829202.325351:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829202.325357:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff8b2e88043480 x1874672518048128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829202.325364:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518048128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829202.325384:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829202.325386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829202.325388:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518048128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829202.325400:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518048256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829202.325410:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829202.325416:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829202/real 1787829202] req@ffff8b2e88042940 x1874672518048256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829202.325423:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518048256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829227 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829202.325428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829202.325430:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829202.325432:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518048256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829202.644709:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829203.156781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829203.156830:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829203.348755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829203.348775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829203.668711:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829204.180753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829204.180892:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829204.372768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829204.372768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829204.692725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829205.204746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829205.204872:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829205.396745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829205.396825:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829205.716752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829206.228775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829206.228827:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829206.420753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829206.420761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829206.740751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829207.252745:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829207.252887:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829207.444716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829207.444721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829207.444810:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829207.444820:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829207.444826:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829207.444837:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2213 00000100:00080000:1.0:1787829207.444850:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829207.444858:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829207.444925:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518048384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829207.444968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829207.444972:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829207.444974:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829207.445006:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2213 00000100:00080000:1.0:1787829207.445009:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829207.445014:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829207.445025:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829207.445039:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518048512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829207.445040:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518048384:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829207.445090:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829207.445094:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829207.445096:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829207.445100:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2213 00000100:00080000:1.0:1787829207.445103:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829207.445108:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829207.445131:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829207.445149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518048640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829207.445162:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829207.445180:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829207.445201:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829207.445203:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829207.445210:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2213 00000100:00080000:1.0:1787829207.445216:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829207.445229:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829207.445229:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff8b2e88077840 x1874672518048384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829207.445239:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829207.445259:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518048768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829207.445269:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518048384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829207.445270:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829207.445278:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829207.445281:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829207.445289:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2213 00000100:00080000:1.0:1787829207.445304:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829207.445307:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829207.445311:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829207.445327:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829207.445333:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829207.445338:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518048896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829207.445343:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518048384:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829207.445358:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829207.445361:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829207.445363:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829207.445367:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2213 00000100:00080000:1.0:1787829207.445370:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829207.445374:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829207.445378:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829207.445399:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518049024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829207.445415:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518048512:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829207.445440:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829207.445444:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829207.445446:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829207.445453:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2213 00000100:00080000:1.0:1787829207.445458:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829207.445462:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829207.445463:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829207.445483:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829207.445498:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518049152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829207.445508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff8b2e88074780 x1874672518048512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829207.445526:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518048512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829207.445538:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829207.445543:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829207.445547:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518048512:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829207.445556:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518048640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829207.445583:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829207.445594:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff8b2e88076940 x1874672518048640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829207.445608:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518048640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829207.445617:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829207.445644:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829207.445697:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518048640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829207.445704:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518048768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829207.445740:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829207.445750:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff8b2e88074f00 x1874672518048768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829207.445781:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518048768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829207.445794:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829207.445799:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829207.445804:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518048768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829207.445813:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518048896:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829207.445843:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829207.445856:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff8b2e88075680 x1874672518048896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829207.445874:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518048896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829207.445882:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829207.445885:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829207.445890:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518048896:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829207.445919:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518049024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829207.445970:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829207.445981:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff8b2e88075a40 x1874672518049024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829207.445990:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518049024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829207.445996:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829207.445998:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829207.446000:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518049024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829207.446004:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518049152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829207.446041:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829207.446053:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829207/real 1787829207] req@ffff8b2e88077480 x1874672518049152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829207.446066:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518049152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829232 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829207.446096:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829207.446099:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829207.446101:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518049152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829207.764725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829208.276784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829208.276852:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829208.468775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829208.468775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829208.788730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829209.300736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829209.300806:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829209.492729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829209.492730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829209.812762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829210.324741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829210.324808:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829210.516758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829210.516758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829210.836740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829211.348704:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829211.348793:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829211.540729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829211.540739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829211.860744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829212.372764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829212.372803:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829212.564755:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829212.564755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829212.564755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829212.564768:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829212.564774:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829212.564785:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2218 00000100:00080000:1.0:1787829212.564797:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829212.564804:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829212.564846:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518049280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829212.564881:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829212.564885:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829212.564887:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829212.564891:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2218 00000100:00080000:1.0:1787829212.564895:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829212.564900:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829212.564911:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829212.564925:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518049280:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829212.564931:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518049408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829212.564955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829212.564959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829212.564962:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829212.564966:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2218 00000100:00080000:1.0:1787829212.564971:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829212.564975:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829212.564981:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829212.564996:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829212.564998:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518049536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829212.565011:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829212.565015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829212.565018:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829212.565023:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2218 00000100:00080000:1.0:1787829212.565027:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829212.565032:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829212.565047:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829212.565048:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff8b2e88077c00 x1874672518049280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829212.565063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518049664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829212.565074:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518049280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829212.565079:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829212.565088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829212.565093:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829212.565097:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829212.565100:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2218 00000100:00080000:1.0:1787829212.565105:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829212.565107:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829212.565113:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829212.565119:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518049280:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787829212.565121:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829212.565139:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518049792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829212.565152:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829212.565152:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518049408:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829212.565160:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829212.565164:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829212.565169:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2218 00000100:00080000:1.0:1787829212.565173:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829212.565177:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829212.565183:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829212.565190:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829212.565198:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518049920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829212.565210:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff8b2e88077480 x1874672518049408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829212.565224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829212.565230:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829212.565232:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829212.565238:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518049408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829212.565248:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2218 00000100:00080000:3.0:1787829212.565252:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829212.565254:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829212.565261:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829212.565263:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829212.565271:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518049408:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787829212.565273:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829212.565284:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518049536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829212.565292:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518050048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829212.565315:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829212.565333:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff8b2e88075a40 x1874672518049536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.565348:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518049536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.565357:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.565361:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829212.565364:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518049536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829212.565371:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518049664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829212.565394:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829212.565407:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff8b2e88075680 x1874672518049664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.565421:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518049664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.565430:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.565433:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829212.565437:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518049664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829212.565454:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518049792:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829212.565477:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829212.565489:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff8b2e88074f00 x1874672518049792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.565503:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518049792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.565512:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.565515:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829212.565519:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518049792:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829212.565525:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518049920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829212.565546:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829212.565559:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff8b2e88076940 x1874672518049920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.565572:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518049920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.565581:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.565584:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829212.565606:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518049920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829212.565613:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518050048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829212.565634:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829212.565671:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829212/real 1787829212] req@ffff8b2e88074780 x1874672518050048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829212.565686:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518050048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829237 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829212.565695:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829212.565699:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829212.565703:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518050048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829212.884747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829213.396720:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829213.396775:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829213.588803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829213.588820:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829213.908765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829214.420736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829214.420782:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829214.612741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829214.612741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829214.932739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829215.444736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829215.444788:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829215.636738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829215.636826:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829215.956725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829216.468744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829216.468802:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829216.660723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829216.660723:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829216.980752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829217.492725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829217.492776:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829217.684772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829217.684780:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829217.684787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829217.684788:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829217.684807:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829217.684817:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2224 00000100:00080000:1.0:1787829217.684827:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829217.684833:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829217.684878:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518050176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829217.684899:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829217.684902:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829217.684904:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829217.684907:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2224 00000100:00080000:1.0:1787829217.684909:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829217.684912:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829217.684921:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829217.684941:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518050304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829217.684949:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829217.684952:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829217.684953:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518050176:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829217.684954:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829217.684962:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2224 00000100:00080000:1.0:1787829217.684966:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829217.684969:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829217.684972:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829217.684981:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518050432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829217.684987:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829217.684990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829217.684991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829217.684993:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2224 00000100:00080000:1.0:1787829217.684995:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829217.684998:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829217.685001:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829217.685008:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518050560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829217.685032:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829217.685033:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829217.685039:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829217.685041:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829217.685047:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2224 00000100:00080000:1.0:1787829217.685054:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829217.685059:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829217.685064:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829217.685076:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff8b2e88077840 x1874672518050176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829217.685080:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518050688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829217.685113:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518050176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829217.685115:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829217.685124:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829217.685127:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829217.685133:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2224 00000100:00080000:3.0:1787829217.685133:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829217.685141:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829217.685153:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829217.685159:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829217.685160:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518050176:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787829217.685167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829217.685178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518050816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829217.685185:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518050304:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829217.685189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829217.685193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829217.685195:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829217.685198:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2224 00000100:00080000:1.0:1787829217.685203:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829217.685206:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829217.685210:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787829217.685213:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829217.685224:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff8b2e88076580 x1874672518050304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829217.685226:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518050944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829217.685268:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518050304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.685280:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.685283:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829217.685286:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518050304:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829217.685291:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518050432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829217.685309:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829217.685318:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff8b2e88076d00 x1874672518050432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829217.685326:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518050432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.685332:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.685342:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829217.685345:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518050432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829217.685349:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518050560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829217.685361:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829217.685368:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff8b2e880770c0 x1874672518050560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829217.685375:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518050560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.685381:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.685383:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829217.685385:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518050560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829217.685388:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518050688:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829217.685398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829217.685404:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff8b2e88074000 x1874672518050688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829217.685411:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518050688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.685416:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.685418:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829217.685420:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518050688:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829217.685432:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518050816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829217.685452:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829217.685458:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff8b2e880761c0 x1874672518050816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829217.685465:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518050816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.685470:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.685472:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829217.685474:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518050816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829217.685477:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518050944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829217.685486:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829217.685491:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829217/real 1787829217] req@ffff8b2e88075e00 x1874672518050944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829217.685498:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518050944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829242 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829217.685503:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829217.685505:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829217.685508:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518050944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829218.004719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829218.516817:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829218.516880:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829218.708766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829218.708782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829219.028753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829219.540729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829219.540807:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829219.732783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829219.732783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829220.052728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829220.564718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829220.564800:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829220.756747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829220.756747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829221.076770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829221.588712:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829221.588844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829221.780736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829221.780757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829222.100748:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829222.612740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829222.612854:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829222.804733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829222.804804:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829222.804809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829222.804816:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829222.804822:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829222.804832:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2229 00000100:00080000:2.0:1787829222.804844:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829222.804853:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829222.804893:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518051072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829222.804922:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829222.804926:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829222.804928:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829222.804934:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2229 00000100:00080000:2.0:1787829222.804939:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829222.804943:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829222.804951:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518051072:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787829222.804955:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829222.804972:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518051200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829222.804985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829222.804989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829222.804991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829222.804995:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2229 00000100:00080000:2.0:1787829222.805001:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829222.805006:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829222.805011:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829222.805018:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829222.805026:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518051328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829222.805038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829222.805042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829222.805044:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829222.805048:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2229 00000100:00080000:2.0:1787829222.805052:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829222.805057:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829222.805063:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829222.805064:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff8b2e8801da40 x1874672518051072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829222.805082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518051456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829222.805084:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518051072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.805114:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829222.805116:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829222.805119:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829222.805125:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787829222.805127:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518051072:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829222.805130:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829222.805138:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2229 00000100:00080000:2.0:1787829222.805143:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829222.805147:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829222.805153:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829222.805153:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518051200:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787829222.805173:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518051584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829222.805185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829222.805190:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829222.805192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829222.805195:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829222.805201:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2229 00000100:00080000:2.0:1787829222.805206:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829222.805211:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829222.805216:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff8b2e8801fc00 x1874672518051200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829222.805218:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829222.805236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518051712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829222.805239:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518051200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.805264:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.805269:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829222.805276:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518051200:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829222.805284:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829222.805287:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518051328:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787829222.805292:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829222.805295:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829222.805301:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2229 00000100:00080000:2.0:1787829222.805307:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829222.805311:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829222.805314:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829222.805322:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829222.805328:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff8b2e8801f480 x1874672518051328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829222.805339:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518051840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829222.805348:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518051328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.805362:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.805366:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829222.805371:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518051328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829222.805376:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518051456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829222.805393:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829222.805401:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff8b2e8801cf00 x1874672518051456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829222.805409:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518051456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.805419:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.805421:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829222.805423:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518051456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829222.805435:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518051584:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829222.805446:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829222.805452:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff8b2e8801c000 x1874672518051584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829222.805460:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518051584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.805466:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.805467:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829222.805469:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518051584:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829222.805473:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518051712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829222.805482:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829222.805488:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff8b2e8801e580 x1874672518051712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829222.805495:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518051712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.805500:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.805501:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829222.805504:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518051712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829222.805508:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518051840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829222.805535:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829222.805543:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829222/real 1787829222] req@ffff8b2e8801e940 x1874672518051840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829222.805551:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518051840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829247 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829222.805557:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829222.805559:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829222.805561:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518051840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829223.124776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829223.636703:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829223.636786:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829223.828761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829223.828761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829224.148785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829224.660775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829224.660844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829224.852716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829224.852718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829225.172777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829225.684765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829225.684829:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829225.876801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829225.876812:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829226.196761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829226.708732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829226.708843:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829226.900753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829226.900754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829227.220721:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829227.732768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829227.732844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829227.924764:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829227.924765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829227.924781:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829227.924786:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829227.924797:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2234 00000100:00080000:0.0:1787829227.924809:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829227.924813:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829227.924818:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829227.924882:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518051968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829227.924922:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829227.924934:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829227.924936:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829227.924942:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2234 00000100:00080000:0.0:1787829227.924947:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829227.924951:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829227.924960:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829227.924967:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518051968:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829227.924976:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518052096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829227.924990:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829227.924993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829227.924995:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829227.925000:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2234 00000100:00080000:0.0:1787829227.925004:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829227.925007:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829227.925013:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829227.925036:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518052224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829227.925041:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829227.925051:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829227.925055:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829227.925058:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829227.925063:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2234 00000100:00080000:0.0:1787829227.925069:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829227.925074:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829227.925080:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829227.925094:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff8b2e880403c0 x1874672518051968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829227.925105:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518052352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829227.925118:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518051968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829227.925119:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829227.925128:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829227.925133:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829227.925142:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2234 00000100:00080000:3.0:1787829227.925144:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829227.925148:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829227.925153:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829227.925158:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829227.925164:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518051968:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829227.925166:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829227.925181:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518052480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829227.925190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829227.925194:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829227.925194:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518052096:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829227.925198:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829227.925205:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2234 00000100:00080000:0.0:1787829227.925208:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829227.925211:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829227.925214:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829227.925223:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518052608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829227.925230:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829227.925233:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829227.925234:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829227.925238:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829227.925242:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2234 00000100:00080000:0.0:1787829227.925254:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829227.925257:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829227.925260:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829227.925266:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff8b2e88042940 x1874672518052096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829227.925273:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518052736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829227.925292:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518052096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829227.925304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829227.925308:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829227.925323:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518052096:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829227.925330:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518052224:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829227.925349:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829227.925359:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff8b2e88043480 x1874672518052224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829227.925378:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518052224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829227.925393:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829227.925397:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829227.925401:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518052224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829227.925407:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518052352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829227.925423:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829227.925432:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff8b2e880430c0 x1874672518052352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829227.925444:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518052352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829227.925458:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829227.925461:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829227.925465:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518052352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829227.925493:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518052480:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829227.925518:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829227.925538:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff8b2e88040000 x1874672518052480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829227.925551:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518052480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829227.925566:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829227.925570:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829227.925585:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518052480:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829227.925591:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518052608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829227.925608:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829227.925618:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff8b2e88041a40 x1874672518052608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829227.925630:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518052608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829227.925665:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829227.925669:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829227.925672:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518052608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829227.925678:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518052736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829227.925697:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829227.925708:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829227/real 1787829227] req@ffff8b2e88040f00 x1874672518052736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829227.925721:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518052736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829252 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829227.925729:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829227.925733:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829227.925744:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518052736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829228.244717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829228.756743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829228.756805:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829228.948727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829228.948753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829229.268762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829229.780761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829229.780838:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829229.972779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829229.972784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829230.292734:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829230.804717:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829230.804810:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829230.996719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829230.996719:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829231.316759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829231.828761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829231.828792:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829232.020771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829232.020771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829232.340726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829232.852774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829232.852839:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829233.044735:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829233.044737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829233.044739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829233.044746:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829233.044750:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829233.044759:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2239 00000100:00080000:0.0:1787829233.044769:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829233.044775:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829233.044809:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518052864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829233.044830:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829233.044833:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829233.044834:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829233.044837:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2239 00000100:00080000:0.0:1787829233.044840:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829233.044843:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829233.044852:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829233.044861:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518052864:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829233.044865:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518052992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829233.044874:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829233.044876:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829233.044878:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829233.044880:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2239 00000100:00080000:0.0:1787829233.044882:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829233.044885:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829233.044894:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829233.044902:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518053120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829233.044909:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829233.044911:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829233.044913:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829233.044915:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2239 00000100:00080000:0.0:1787829233.044919:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829233.044919:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829233.044926:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829233.044930:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829233.044938:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518053248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829233.044945:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829233.044947:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829233.044948:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829233.044951:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2239 00000100:00080000:0.0:1787829233.044953:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829233.044956:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829233.044960:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829233.044964:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829233/real 1787829233] req@ffff8b2e88040b40 x1874672518052864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829233.044969:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518053376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829233.044988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829233.044991:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518052864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829233.044995:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829233.045002:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829233.045007:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829233.045009:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2239 00000100:00080000:3.0:1787829233.045013:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829233.045019:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829233.045019:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518052864:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829233.045029:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829233.045036:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829233.045042:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518052992:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829233.045051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518053504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829233.045064:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829233.045064:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829233.045070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829233.045073:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829233.045077:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829233/real 1787829233] req@ffff8b2e88041e00 x1874672518052992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829233.045079:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2239 00000100:00080000:0.0:1787829233.045089:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829233.045094:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518052992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829233.045096:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829233.045105:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829233.045106:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829233.045112:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829233.045115:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518052992:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829233.045124:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518053632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829233.045124:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518053120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829233.045140:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829233.045148:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829233/real 1787829233] req@ffff8b2e88040780 x1874672518053120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829233.045156:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518053120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829233.045162:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829233.045164:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829233.045166:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518053120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829233.045170:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518053248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829233.045178:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829233.045184:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829233/real 1787829233] req@ffff8b2e88042d00 x1874672518053248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829233.045195:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518053248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829233.045201:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829233.045202:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829233.045204:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518053248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829233.045207:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518053376:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829233.045216:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829233.045221:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829233/real 1787829233] req@ffff8b2e880421c0 x1874672518053376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829233.045228:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518053376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829233.045234:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829233.045236:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829233.045239:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518053376:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829233.045255:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518053504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829233.045275:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829233.045286:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829233/real 1787829233] req@ffff8b2e880412c0 x1874672518053504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829233.045299:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518053504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829233.045309:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829233.045311:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829233.045313:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518053504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829233.045317:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518053632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829233.045342:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829233.045353:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829233/real 1787829233] req@ffff8b2e88043840 x1874672518053632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829233.045360:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518053632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829258 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829233.045366:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829233.045368:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829233.045369:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518053632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829233.364784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829233.876817:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829233.876892:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829234.068767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829234.068768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829234.388825:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829234.900736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829234.900796:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829235.092719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829235.092719:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829235.412989:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829235.924753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829235.924850:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829236.116761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829236.116768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829236.436762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829236.948732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829236.948796:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829237.140723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829237.140756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829237.460757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829237.972771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829237.972811:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829238.164732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829238.164745:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829238.164747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829238.164751:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829238.164756:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829238.164766:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2244 00000100:00080000:0.0:1787829238.164778:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829238.164786:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829238.164834:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518053760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829238.164864:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829238.164867:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829238.164868:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829238.164872:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2244 00000100:00080000:0.0:1787829238.164875:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829238.164878:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829238.164886:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829238.164899:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518053888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829238.164906:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829238.164909:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829238.164910:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829238.164912:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2244 00000100:00080000:0.0:1787829238.164914:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829238.164917:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829238.164920:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829238.164928:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518054016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829238.164935:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829238.164940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829238.164940:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518053760:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829238.164944:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829238.164951:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2244 00000100:00080000:0.0:1787829238.164953:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829238.164956:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829238.164959:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829238.164968:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518054144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829238.164974:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829238.164976:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829238.164978:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829238.164982:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2244 00000100:00080000:0.0:1787829238.164984:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829238.164987:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829238.164990:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829238.164998:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518054272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829238.165004:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829238.165010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829238.165013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829238.165015:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829238.165018:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2244 00000100:00080000:0.0:1787829238.165021:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829238.165023:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829238.165026:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829238.165034:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518054400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829238.165050:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829238/real 1787829238] req@ffff8b2e88042580 x1874672518053760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829238.165052:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829238.165061:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829238.165065:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829238.165070:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518053760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829238.165072:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2244 00000100:00080000:0.0:1787829238.165080:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829238.165084:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829238.165086:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829238.165089:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829238.165095:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518053760:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829238.165096:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829238.165101:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518053888:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829238.165113:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518054528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829238.165130:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829238.165145:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829238/real 1787829238] req@ffff8b2e88043840 x1874672518053888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829238.165154:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518053888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829238.165160:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829238.165162:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829238.165164:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518053888:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829238.165183:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518054016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829238.165195:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829238.165201:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829238/real 1787829238] req@ffff8b2e880412c0 x1874672518054016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829238.165214:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518054016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829238.165219:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829238.165221:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829238.165223:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518054016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829238.165227:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518054144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829238.165235:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829238.165241:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829238/real 1787829238] req@ffff8b2e880421c0 x1874672518054144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829238.165248:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518054144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829238.165253:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829238.165255:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829238.165257:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518054144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829238.165260:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518054272:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829238.165269:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829238.165274:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829238/real 1787829238] req@ffff8b2e88042d00 x1874672518054272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829238.165281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518054272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829238.165291:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829238.165293:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829238.165295:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518054272:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829238.165298:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518054400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829238.165307:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829238.165312:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829238/real 1787829238] req@ffff8b2e88040780 x1874672518054400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829238.165319:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518054400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829238.165324:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829238.165326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829238.165328:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518054400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829238.165331:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518054528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829238.165339:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829238.165344:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829238/real 1787829238] req@ffff8b2e88041e00 x1874672518054528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829238.165351:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518054528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829263 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829238.165356:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829238.165358:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829238.165361:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518054528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829238.484774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829238.996726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829238.996782:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829239.188745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829239.188767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829239.508729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829240.020809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829240.020824:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829240.212748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829240.212752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829240.532739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829241.044719:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829241.044780:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829241.236770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829241.236782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829241.556704:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829242.068736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829242.068827:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829242.260711:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829242.260723:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829242.580774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829243.092709:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829243.092809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829243.284738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829243.284761:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829243.284764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829243.284774:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829243.284779:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829243.284789:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2249 00000100:00080000:0.0:1787829243.284802:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829243.284809:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829243.284856:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518054656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829243.284884:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829243.284888:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829243.284889:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829243.284893:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2249 00000100:00080000:0.0:1787829243.284896:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829243.284900:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829243.284911:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829243.284920:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518054656:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829243.284926:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518054784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829243.284942:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829243.284946:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829243.284949:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829243.284954:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2249 00000100:00080000:0.0:1787829243.284959:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829243.284965:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829243.284972:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829243.284990:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518054912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829243.284997:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829243.285012:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829243.285015:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829243.285017:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829243.285023:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2249 00000100:00080000:0.0:1787829243.285027:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829243.285047:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829243.285054:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829243.285058:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff8b2e88040b40 x1874672518054656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829243.285069:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518055040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829243.285080:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518054656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829243.285083:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829243.285098:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829243.285103:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829243.285103:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829243.285109:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829243.285118:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2249 00000100:00100000:3.0:1787829243.285119:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518054656:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829243.285146:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829243.285154:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829243.285160:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518054784:192.168.203.101@tcp:38:kworker.0 00000008:00100000:0.0:1787829243.285161:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829243.285189:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518055168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829243.285201:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829243.285201:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829243.285211:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829243.285214:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829243.285218:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2249 00000100:00080000:0.0:1787829243.285224:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829243.285245:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff8b2e88040f00 x1874672518054784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829243.285246:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829243.285254:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829243.285270:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518055296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829243.285281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518054784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829243.285283:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829243.285294:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829243.285297:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829243.285305:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2249 00000100:00080000:0.0:1787829243.285312:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829243.285318:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829243.285327:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829243.285329:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829243.285337:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829243.285338:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518055424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829243.285348:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518054784:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829243.285383:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518054912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829243.285426:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829243.285442:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff8b2e88041a40 x1874672518054912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829243.285455:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518054912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829243.285462:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829243.285464:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829243.285467:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518054912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829243.285472:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518055040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829243.285488:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829243.285496:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff8b2e88040000 x1874672518055040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829243.285503:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518055040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829243.285509:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829243.285511:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829243.285513:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518055040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829243.285526:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518055168:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829243.285537:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829243.285544:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff8b2e880430c0 x1874672518055168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829243.285551:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518055168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829243.285556:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829243.285558:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829243.285560:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518055168:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829243.285564:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518055296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829243.285574:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829243.285579:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff8b2e88043480 x1874672518055296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829243.285605:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518055296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829243.285610:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829243.285612:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829243.285614:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518055296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829243.285618:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518055424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829243.285629:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829243.285635:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829243/real 1787829243] req@ffff8b2e88042940 x1874672518055424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829243.285642:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518055424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829268 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829243.285649:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829243.285651:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829243.285653:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518055424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829243.604734:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829244.116728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829244.116791:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829244.308734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829244.308748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829244.628788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829245.140701:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829245.140870:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829245.332762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829245.332762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829245.652739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829246.164714:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829246.164793:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829246.356704:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829246.356741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829246.676802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829247.188764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829247.188822:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829247.380717:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829247.380718:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829247.700780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829248.212748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829248.212865:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829248.404752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829248.404752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829248.404779:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.404786:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829248.404802:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.404812:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2254 00000100:00080000:1.0:1787829248.404829:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.404836:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829248.404876:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518055552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829248.404927:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.404931:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829248.404933:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.404937:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2254 00000100:00080000:1.0:1787829248.404942:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.404963:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829248.404973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829248.404987:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518055552:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829248.404990:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518055680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829248.405003:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.405006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829248.405008:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.405011:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2254 00000100:00080000:1.0:1787829248.405014:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.405018:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829248.405022:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829248.405032:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518055808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829248.405040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.405043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829248.405044:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.405047:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2254 00000100:00080000:1.0:1787829248.405049:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.405055:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829248.405057:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787829248.405060:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829248.405069:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518055936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829248.405077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.405079:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829248.405081:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.405084:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2254 00000100:00080000:1.0:1787829248.405088:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.405090:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829248.405094:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829248.405102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518056064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829248.405110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.405114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829248.405115:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829248.405122:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff8b2e88074b40 x1874672518055552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829248.405123:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2254 00000100:00080000:1.0:1787829248.405129:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.405136:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829248.405149:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829248.405152:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518055552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829248.405162:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518056192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829248.405166:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829248.405178:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829248.405179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829248.405185:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829248.405186:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518055552:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829248.405187:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829248.405196:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2254 00000100:00080000:1.0:1787829248.405200:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829248.405203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829248.405217:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829248.405225:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518056320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829248.405230:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518055680:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829248.405264:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829248.405285:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff8b2e88075e00 x1874672518055680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829248.405296:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518055680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829248.405304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829248.405306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829248.405309:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518055680:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829248.405314:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518055808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829248.405328:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829248.405356:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff8b2e880761c0 x1874672518055808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829248.405379:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518055808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829248.405388:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829248.405393:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829248.405397:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518055808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829248.405405:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518055936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829248.405432:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829248.405448:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff8b2e88074000 x1874672518055936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829248.405461:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518055936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829248.405473:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829248.405476:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829248.405479:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518055936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829248.405484:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518056064:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829248.405501:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829248.405509:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff8b2e880770c0 x1874672518056064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829248.405518:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518056064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829248.405525:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829248.405528:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829248.405530:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518056064:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829248.405535:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518056192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829248.405546:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829248.405564:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff8b2e88076d00 x1874672518056192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829248.405574:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518056192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829248.405590:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829248.405593:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829248.405596:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518056192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829248.405619:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518056320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829248.405681:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829248.405695:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829248/real 1787829248] req@ffff8b2e88076580 x1874672518056320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829248.405706:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518056320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829273 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829248.405713:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829248.405715:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829248.405718:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518056320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829248.724809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829249.236775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829249.236831:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829249.428777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829249.428809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829249.748827:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787829250.260726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829250.260827:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829250.452739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829250.452741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829250.772821:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829251.284794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829251.284837:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829251.476757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829251.476758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829251.796766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829252.308722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829252.308847:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829252.500727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829252.500732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829252.820746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829253.332724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829253.332779:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829253.524735:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829253.524737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829253.524737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829253.524747:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829253.524751:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829253.524760:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2259 00000100:00080000:1.0:1787829253.524770:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829253.524778:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829253.524814:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518056448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829253.524839:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829253.524842:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829253.524843:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829253.524847:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2259 00000100:00080000:1.0:1787829253.524849:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829253.524853:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829253.524862:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829253.524890:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518056448:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829253.524894:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518056576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829253.524905:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829253.524908:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829253.524910:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829253.524913:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2259 00000100:00080000:1.0:1787829253.524916:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829253.524919:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829253.524924:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829253.524934:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518056704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829253.524942:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829253.524945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829253.524946:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829253.524951:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2259 00000800:00000100:3.0:1787829253.524956:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829253.524958:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829253.524964:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829253.524969:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829253.524980:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518056832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829253.524987:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829253.524990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829253.524991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829253.524995:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2259 00000100:00080000:1.0:1787829253.524998:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829253.525004:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829253.525005:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff8b2e88077840 x1874672518056448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829253.525008:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829253.525023:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518056960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829253.525032:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829253.525034:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518056448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829253.525035:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829253.525039:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829253.525047:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2259 00000100:00080000:1.0:1787829253.525053:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829253.525077:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829253.525081:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829253.525082:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829253.525093:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829253.525094:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518057088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829253.525105:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518056448:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829253.525108:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829253.525113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829253.525115:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829253.525119:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2259 00000100:00080000:1.0:1787829253.525122:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829253.525125:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829253.525128:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829253.525134:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518056576:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829253.525139:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518057216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829253.525186:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829253.525206:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff8b2e88074780 x1874672518056576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829253.525240:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518056576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829253.525251:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829253.525254:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829253.525257:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518056576:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829253.525263:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518056704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829253.525281:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829253.525291:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff8b2e88076940 x1874672518056704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829253.525314:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518056704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829253.525324:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829253.525327:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829253.525329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518056704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829253.525334:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518056832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829253.525351:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829253.525360:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff8b2e88074f00 x1874672518056832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829253.525369:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518056832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829253.525375:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829253.525377:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829253.525380:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518056832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829253.525384:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518056960:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829253.525395:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829253.525402:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff8b2e88075680 x1874672518056960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829253.525410:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518056960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829253.525416:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829253.525418:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829253.525421:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518056960:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829253.525425:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518057088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829253.525436:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829253.525442:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff8b2e88075a40 x1874672518057088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829253.525456:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518057088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829253.525462:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829253.525464:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829253.525466:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518057088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829253.525482:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518057216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829253.525494:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829253.525501:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829253/real 1787829253] req@ffff8b2e88077480 x1874672518057216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829253.525509:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518057216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829278 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829253.525515:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829253.525517:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829253.525519:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518057216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829253.844742:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829254.356746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829254.356827:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829254.548726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829254.548726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829254.868749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829255.380794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829255.380884:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829255.572736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829255.572737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829255.892776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829256.404770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829256.404881:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829256.596713:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829256.596724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829256.916729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829257.428728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829257.428854:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829257.620743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829257.620743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829257.940739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829258.452734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829258.452799:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829258.644762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829258.644763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829258.644773:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.644779:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829258.644784:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.644792:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2265 00000100:00080000:1.0:1787829258.644801:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.644807:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829258.644854:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518057344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.644888:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.644891:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829258.644893:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.644897:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2265 00000100:00080000:1.0:1787829258.644899:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.644902:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829258.644911:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829258.644914:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518057344:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829258.644924:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518057472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.644933:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.644936:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829258.644937:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.644940:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2265 00000100:00080000:1.0:1787829258.644942:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.644954:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829258.644958:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829258.644966:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518057600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.644973:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.644975:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829258.644976:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.644979:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2265 00000100:00080000:1.0:1787829258.644981:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.644983:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829258.644986:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829258.644994:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518057728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.645001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.645003:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829258.645005:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.645008:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2265 00000100:00080000:1.0:1787829258.645010:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.645013:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829258.645017:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829258.645018:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518057472:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829258.645029:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518057856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.645039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.645041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829258.645042:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.645048:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2265 00000100:00100000:3.0:1787829258.645048:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518057600:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829258.645056:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829258.645060:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829258.645063:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829258.645073:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518057728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829258.645075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518057984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829258.645084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829258.645087:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829258.645088:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829258.645092:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2265 00000100:00080000:1.0:1787829258.645095:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829258.645098:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518057856:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829258.645101:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829258.645107:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829258.645118:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518058112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829258.645124:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518057984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829258.645154:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518058112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829258.964768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829259.476808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829259.476957:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829259.668809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829259.668956:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829259.988753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829260.500760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829260.500828:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829260.692790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829260.692865:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829261.012718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829261.524790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829261.524905:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829261.716736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829261.716966:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:1.0:1787829261.716980:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787829261.717024:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829261.717038:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829261.717081:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829261.717088:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829261.717095:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829261.717106:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829261.717129:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829261.717139:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829261] req@ffff8b2e88074f00 x1874672518057856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787829261.717140:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829261.717171:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518057856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829261.717185:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829261.717190:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829261.717194:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518057856:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829261.717202:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829261] req@ffff8b2e88076940 x1874672518057984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829261.717215:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518057984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829261.717223:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829261.717228:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829261.717231:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518057984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829261.717237:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829261] req@ffff8b2e88077480 x1874672518057472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829261.717270:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518057472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829261.717280:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829261.717285:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829261.717288:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518057472:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829261.717300:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829261] req@ffff8b2e88075a40 x1874672518057600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829261.717314:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518057600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829261.717322:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829261.717326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829261.717329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518057600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829261.717335:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829261] req@ffff8b2e88075680 x1874672518057728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829261.717346:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518057728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829261.717355:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829261.717359:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829261.717362:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518057728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829261.717369:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829261] req@ffff8b2e88077c00 x1874672518057344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829261.717382:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518057344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829261.717390:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829261.717393:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829261.717397:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518057344:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829261.717428:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829258/real 1787829261] req@ffff8b2e88074780 x1874672518058112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829261.717441:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518058112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829283 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829261.717449:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829261.717453:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829261.717457:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518058112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829262.036800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829262.548712:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829262.548861:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829262.740744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829262.740805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829263.060724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829263.572744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829263.572801:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829263.764763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829263.764802:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829263.764812:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829263.764819:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829263.764831:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2270 00000100:00100000:2.0:1787829263.764843:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829263.764846:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829263.764859:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829263.764901:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518058240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829263.764958:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829263.764965:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829263.764969:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829263.764978:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2270 00000100:00080000:1.0:1787829263.764985:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829263.765007:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829263.765033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829263.765052:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518058368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829263.765073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829263.765077:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829263.765079:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829263.765084:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2270 00000100:00080000:1.0:1787829263.765089:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829263.765095:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829263.765101:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829263.765117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518058496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829263.765129:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518058240:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829263.765137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829263.765144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829263.765146:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829263.765153:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2270 00000100:00080000:1.0:1787829263.765158:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829263.765163:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829263.765170:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829263.765187:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518058624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829263.765200:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829263.765204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829263.765207:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829263.765215:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2270 00000800:00000100:3.0:1787829263.765217:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829263.765226:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829263.765257:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829263.765265:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829263.765284:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518058752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829263.765299:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829263.765304:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829263.765309:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829263.765309:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff8b2e88077840 x1874672518058240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829263.765322:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2270 00000100:00080000:1.0:1787829263.765333:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829263.765343:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829263.765357:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829263.765375:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518058240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829263.765392:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518058880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829263.765410:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829263.765415:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829263.765417:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829263.765423:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829263.765430:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829263.765436:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2270 00000100:00100000:3.0:1787829263.765442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518058240:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829263.765447:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829263.765458:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518058368:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829263.765460:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829263.765474:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829263.765496:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518059008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829263.765514:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829263.765545:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff8b2e88076580 x1874672518058368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829263.765565:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518058368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829263.765578:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829263.765583:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829263.765587:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518058368:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829263.765618:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518058496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829263.765681:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829263.765701:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff8b2e88076d00 x1874672518058496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829263.765846:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518058496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829263.765880:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829263.765886:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829263.765891:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518058496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829263.765900:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518058624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829263.765934:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829263.765954:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff8b2e880770c0 x1874672518058624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829263.765972:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518058624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829263.765984:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829263.765991:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829263.765996:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518058624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829263.766004:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518058752:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829263.766032:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829263.766048:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff8b2e88074000 x1874672518058752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829263.766063:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518058752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829263.766075:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829263.766090:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829263.766095:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518058752:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829263.766103:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518058880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829263.766137:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829263.766153:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff8b2e880761c0 x1874672518058880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829263.766169:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518058880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829263.766179:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829263.766182:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829263.766186:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518058880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829263.766194:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518059008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829263.766219:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829263.766253:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829263/real 1787829263] req@ffff8b2e88075e00 x1874672518059008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829263.766271:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518059008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829288 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829263.766281:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829263.766285:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829263.766290:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518059008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829264.084730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829264.596757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829264.596827:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829264.788737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829264.788880:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829265.108757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829265.620722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829265.620828:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829265.812728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829265.812728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829266.132714:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829266.644801:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829266.644956:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829266.836731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829266.836731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829267.156769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829267.668749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829267.668807:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829267.860731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829267.860774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829268.180733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829268.692783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829268.692885:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829268.884775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829268.884795:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829268.884798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829268.884805:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829268.884810:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829268.884821:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2275 00000100:00080000:2.0:1787829268.884834:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829268.884843:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829268.884892:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518059136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829268.884925:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829268.884928:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829268.884930:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829268.884935:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2275 00000100:00080000:2.0:1787829268.884940:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829268.884945:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829268.884956:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829268.884962:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518059136:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829268.884975:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518059264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829268.884985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829268.884989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829268.884991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829268.884994:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2275 00000100:00080000:2.0:1787829268.884997:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829268.885000:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829268.885005:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829268.885014:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518059392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829268.885023:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829268.885026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829268.885027:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829268.885031:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2275 00000100:00080000:2.0:1787829268.885034:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829268.885037:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829268.885042:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829268.885048:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829268.885058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518059520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829268.885066:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829268.885069:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829268.885070:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829268.885073:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2275 00000100:00080000:2.0:1787829268.885076:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829268.885080:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829268.885084:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829268.885090:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff8b2e8801e940 x1874672518059136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829268.885098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518059648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829268.885111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829268.885116:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518059136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829268.885117:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829268.885122:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829268.885133:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2275 00000100:00080000:3.0:1787829268.885138:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829268.885139:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829268.885147:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829268.885150:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:2.0:1787829268.885153:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829268.885156:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518059136:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829268.885168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518059776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829268.885177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829268.885180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829268.885182:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829268.885187:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518059264:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829268.885188:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2275 00000100:00080000:2.0:1787829268.885214:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829268.885236:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829268.885241:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829268.885254:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829268.885258:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518059904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829268.885267:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff8b2e8801e580 x1874672518059264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.885299:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518059264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.885310:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.885314:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829268.885318:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518059264:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829268.885323:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518059392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829268.885343:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829268.885352:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff8b2e8801c000 x1874672518059392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.885362:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518059392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.885369:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.885371:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829268.885374:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518059392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829268.885378:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518059520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829268.885390:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829268.885397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff8b2e8801cf00 x1874672518059520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.885406:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518059520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.885413:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.885415:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829268.885418:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518059520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829268.885422:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518059648:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829268.885433:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829268.885440:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff8b2e8801f480 x1874672518059648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.885462:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518059648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.885473:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.885477:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829268.885481:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518059648:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829268.885488:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518059776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829268.885514:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829268.885542:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff8b2e8801fc00 x1874672518059776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.885550:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518059776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.885556:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.885558:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829268.885560:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518059776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829268.885573:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518059904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829268.885602:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829268.885610:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829268/real 1787829268] req@ffff8b2e8801da40 x1874672518059904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829268.885620:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518059904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829293 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829268.885631:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829268.885634:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829268.885636:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518059904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829269.204765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829269.716731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829269.716879:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829269.908773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829269.908808:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829270.228734:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829270.740769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829270.740872:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829270.932784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829270.932790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829271.252719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829271.764781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829271.764891:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829271.956756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829271.956793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829272.276745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829272.788724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829272.788791:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829272.980753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829272.980753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829273.300719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829273.812756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829273.812824:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829274.004716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829274.004729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829274.004739:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829274.004747:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829274.004753:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829274.004763:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2280 00000100:00080000:2.0:1787829274.004774:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829274.004782:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829274.004822:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518060032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829274.004847:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829274.004850:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829274.004853:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829274.004857:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2280 00000100:00080000:2.0:1787829274.004862:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829274.004866:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829274.004876:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829274.004879:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518060032:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829274.004893:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518060160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829274.004906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829274.004909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829274.004911:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829274.004915:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2280 00000100:00080000:2.0:1787829274.004927:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829274.004931:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829274.004936:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787829274.004940:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829274.004955:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518060288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829274.004965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829274.004968:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829274.004970:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829274.004975:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2280 00000100:00080000:2.0:1787829274.004980:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829274.004981:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829274/real 1787829274] req@ffff8b2e8801c3c0 x1874672518060032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829274.004987:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829274.005002:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518060032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829274.005006:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829274.005010:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829274.005015:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829274.005018:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518060032:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829274.005024:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518060416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829274.005040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829274.005040:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518060160:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829274.005048:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829274.005051:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829274.005057:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2280 00000100:00080000:2.0:1787829274.005062:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829274.005063:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829274.005071:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829274.005077:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829274/real 1787829274] req@ffff8b2e8801cb40 x1874672518060160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829274.005078:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829274.005093:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518060160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829274.005098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518060544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829274.005106:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829274.005113:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829274.005114:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829274.005120:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518060160:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829274.005125:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518060288:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787829274.005128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829274.005135:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829274.005143:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2280 00000800:00000100:3.0:1787829274.005144:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829274.005153:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829274.005156:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829274/real 1787829274] req@ffff8b2e8801de00 x1874672518060288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829274.005184:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829274.005190:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829274.005207:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518060672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829274.005209:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518060288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829274.005235:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829274.005237:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829274.005240:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829274.005243:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829274.005247:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518060288:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787829274.005248:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829274.005257:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2280 00000100:00100000:3.0:1787829274.005260:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518060416:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787829274.005262:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829274.005268:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829274.005272:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829274.005282:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518060800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829274.005285:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829274.005299:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829274/real 1787829274] req@ffff8b2e8801ed00 x1874672518060416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829274.005310:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518060416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829274.005318:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829274.005320:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829274.005325:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518060416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829274.005343:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518060544:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829274.005365:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829274.005378:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829274/real 1787829274] req@ffff8b2e8801c780 x1874672518060544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829274.005388:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518060544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829274.005394:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829274.005396:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829274.005398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518060544:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829274.005402:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518060672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829274.005417:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829274.005423:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829274/real 1787829274] req@ffff8b2e8801f0c0 x1874672518060672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829274.005431:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518060672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829274.005436:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829274.005438:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829274.005440:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518060672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829274.005443:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518060800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829274.005452:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829274.005458:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829274/real 1787829274] req@ffff8b2e8801d680 x1874672518060800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829274.005465:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518060800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829299 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829274.005471:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829274.005472:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829274.005474:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518060800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829274.324758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787829274.836845:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829274.836871:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787829275.028737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829275.028746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829275.348765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829275.860744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829275.860810:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829276.052735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829276.052735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829276.372741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829276.884719:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829276.884807:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829277.076708:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829277.076755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829277.396790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829277.908744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829277.908832:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829278.100762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829278.100762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829278.420708:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829278.932761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829278.932839:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829279.124801:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829279.124818:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829279.124823:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829279.124830:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829279.124838:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2285 00000100:00080000:2.0:1787829279.124853:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829279.124863:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829279.124880:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829279.124917:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518060928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829279.124947:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829279.124953:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829279.124956:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829279.124980:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2285 00000100:00080000:2.0:1787829279.124987:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829279.124993:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829279.125001:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518060928:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787829279.125006:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829279.125030:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518061056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829279.125044:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829279.125048:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829279.125050:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829279.125053:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2285 00000100:00080000:2.0:1787829279.125057:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829279.125061:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829279.125076:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829279.125081:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829279.125093:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518061184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829279.125103:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829279.125106:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829279.125108:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829279.125111:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2285 00000100:00080000:2.0:1787829279.125113:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829279.125116:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829279.125120:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829279.125152:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518061312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829279.125175:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829279/real 1787829279] req@ffff8b2e8801d680 x1874672518060928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829279.125224:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829279.125228:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829279.125231:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829279.125237:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518060928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829279.125240:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2285 00000100:00080000:2.0:1787829279.125247:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829279.125257:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829279.125270:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829279.125273:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829279.125282:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829279.125289:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518061440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829279.125296:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518060928:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829279.125310:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829279.125314:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829279.125316:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829279.125321:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2285 00000100:00080000:2.0:1787829279.125324:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829279.125328:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829279.125332:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829279.125342:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518061056:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787829279.125345:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518061568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829279.125363:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829279.125366:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829279.125368:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829279.125396:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2285 00000100:00080000:2.0:1787829279.125401:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829279.125405:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829279.125410:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829279.125417:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829279.125427:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829279/real 1787829279] req@ffff8b2e8801f0c0 x1874672518061056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829279.125432:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518061696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829279.125458:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518061056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829279.125469:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829279.125473:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829279.125477:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518061056:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829279.125483:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518061184:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829279.125500:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829279.125508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829279/real 1787829279] req@ffff8b2e8801c780 x1874672518061184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829279.125518:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518061184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829279.125536:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829279.125539:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829279.125544:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518061184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829279.125551:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518061312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829279.125589:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829279.125602:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829279/real 1787829279] req@ffff8b2e8801ed00 x1874672518061312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829279.125616:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518061312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829279.125625:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829279.125629:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829279.125633:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518061312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829279.125640:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518061440:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829279.125708:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829279.125726:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829279/real 1787829279] req@ffff8b2e8801de00 x1874672518061440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829279.125742:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518061440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829279.125753:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829279.125757:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829279.125761:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518061440:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829279.125787:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518061568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829279.125815:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829279.125831:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829279/real 1787829279] req@ffff8b2e8801cb40 x1874672518061568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829279.125847:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518061568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829279.125860:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829279.125865:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829279.125871:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518061568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829279.125880:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518061696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829279.125907:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829279.125919:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829279/real 1787829279] req@ffff8b2e8801c3c0 x1874672518061696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829279.125930:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518061696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829304 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829279.125939:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829279.125941:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829279.125944:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518061696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829279.444768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829279.956747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829279.956811:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829280.148826:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829280.148826:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829280.468708:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829280.980753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829280.980818:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829281.172721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829281.172728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829281.492707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829282.004738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829282.004863:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829282.196772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829282.196773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829282.516708:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829283.028736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829283.028844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829283.220711:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829283.220720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829283.540763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829284.052726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829284.052793:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829284.244748:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829284.244748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829284.244749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829284.244761:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829284.244766:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829284.244776:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2290 00000100:00080000:2.0:1787829284.244788:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829284.244796:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829284.244856:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518061824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829284.244886:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829284.244890:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829284.244892:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829284.244896:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2290 00000100:00080000:2.0:1787829284.244916:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829284.244921:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829284.244933:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829284.244943:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518061824:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829284.244952:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518061952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829284.244975:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829284.244980:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829284.244982:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829284.244987:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2290 00000100:00080000:2.0:1787829284.245002:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829284.245007:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829284.245030:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829284.245030:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829284.245048:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518062080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829284.245060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829284.245064:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829284.245067:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829284.245071:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2290 00000100:00080000:2.0:1787829284.245077:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829284.245080:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff8b2e8801da40 x1874672518061824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829284.245086:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829284.245127:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829284.245133:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518061824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829284.245143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518062208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829284.245149:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829284.245152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829284.245157:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829284.245158:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829284.245160:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829284.245169:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2290 00000100:00100000:3.0:1787829284.245171:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518061824:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829284.245173:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829284.245181:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829284.245185:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829284.245193:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518062336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829284.245202:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829284.245205:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518061952:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829284.245206:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829284.245209:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829284.245215:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2290 00000100:00080000:2.0:1787829284.245217:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829284.245219:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829284.245222:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829284.245232:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518062464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829284.245238:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829284.245241:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829284.245242:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829284.245247:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829284.245248:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2290 00000100:00080000:2.0:1787829284.245254:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829284.245257:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829284.245260:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829284.245264:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff8b2e8801fc00 x1874672518061952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829284.245270:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518062592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829284.245285:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518061952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829284.245308:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.245312:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829284.245315:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518061952:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829284.245320:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518062080:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829284.245335:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829284.245342:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff8b2e8801f480 x1874672518062080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829284.245355:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518062080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829284.245363:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.245367:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829284.245370:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518062080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829284.245377:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518062208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829284.245398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829284.245409:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff8b2e8801cf00 x1874672518062208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829284.245417:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518062208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829284.245423:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.245425:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829284.245428:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518062208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829284.245440:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518062336:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829284.245453:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829284.245460:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff8b2e8801c000 x1874672518062336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829284.245468:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518062336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829284.245474:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.245475:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829284.245477:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518062336:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829284.245481:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518062464:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829284.245491:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829284.245496:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff8b2e8801e580 x1874672518062464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829284.245504:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518062464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829284.245509:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.245511:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829284.245513:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518062464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829284.245517:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518062592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829284.245526:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829284.245531:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829284/real 1787829284] req@ffff8b2e8801e940 x1874672518062592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829284.245538:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518062592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829309 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829284.245543:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829284.245544:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829284.245546:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518062592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829284.564713:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829285.076765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829285.076876:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829285.268758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829285.268767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829285.588775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829286.100774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829286.100892:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829286.292761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829286.292761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829286.612795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829287.124758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829287.124820:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829287.316749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829287.316762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829287.636783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829288.148826:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829288.148896:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829288.340720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829288.340726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829288.660744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829289.172722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829289.172790:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829289.364747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829289.364770:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829289.364775:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787829289.364795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829289.364797:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829289.364810:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2295 00000100:00080000:0.0:1787829289.364821:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829289.364829:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829289.364866:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518062720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829289.364892:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829289.364895:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829289.364897:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829289.364900:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2295 00000100:00080000:0.0:1787829289.364903:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829289.364906:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829289.364916:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829289.364928:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518062848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829289.364938:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518062720:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829289.364941:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829289.364949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829289.364950:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829289.364954:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2295 00000100:00080000:0.0:1787829289.364957:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829289.364960:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829289.364964:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829289.364975:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518062976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829289.364984:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829289.364987:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829289.364988:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829289.364991:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2295 00000100:00080000:0.0:1787829289.365002:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829289.365008:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829289.365009:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829289.365017:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829289.365027:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518063104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829289.365035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829289.365038:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829289.365039:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829289.365043:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2295 00000100:00080000:0.0:1787829289.365045:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829289.365049:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829289.365072:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829289.365078:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff8b2e880403c0 x1874672518062720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829289.365084:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518063232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829289.365128:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829289.365130:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518062720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829289.365131:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829289.365135:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829289.365142:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2295 00000100:00080000:0.0:1787829289.365148:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829289.365151:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829289.365153:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829289.365159:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829289.365159:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829289.365166:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518062720:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829289.365176:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518063360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829289.365183:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829289.365185:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829289.365186:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829289.365188:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2295 00000100:00080000:0.0:1787829289.365191:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829289.365195:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829289.365196:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518062848:192.168.203.101@tcp:38:kworker.0 00000008:00100000:0.0:1787829289.365199:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829289.365209:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518063488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829289.365232:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829289.365248:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff8b2e88042940 x1874672518062848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829289.365272:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518062848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829289.365278:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829289.365281:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829289.365283:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518062848:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829289.365288:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518062976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829289.365303:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829289.365310:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff8b2e88043480 x1874672518062976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829289.365318:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518062976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829289.365323:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829289.365325:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829289.365327:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518062976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829289.365332:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518063104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829289.365341:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829289.365347:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff8b2e880430c0 x1874672518063104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829289.365354:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518063104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829289.365359:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829289.365361:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829289.365363:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518063104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829289.365366:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518063232:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829289.365376:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829289.365391:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff8b2e88040000 x1874672518063232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829289.365398:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518063232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829289.365403:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829289.365405:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829289.365407:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518063232:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829289.365410:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518063360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829289.365436:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829289.365442:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff8b2e88041a40 x1874672518063360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829289.365451:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518063360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829289.365456:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829289.365458:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829289.365460:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518063360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829289.365473:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518063488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829289.365484:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829289.365490:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829289/real 1787829289] req@ffff8b2e88040f00 x1874672518063488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829289.365498:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518063488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829314 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829289.365504:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829289.365512:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829289.365514:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518063488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829289.684766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829290.196786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829290.196858:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829290.388771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829290.388771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829290.708716:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829291.220743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829291.220833:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829291.412777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829291.412777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829291.732763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829292.244750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829292.244839:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829292.436759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829292.436760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829292.756793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829293.268723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829293.268833:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829293.460710:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829293.460717:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829293.780750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829294.292750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829294.292825:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829294.484716:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829294.484725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829294.484726:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829294.484739:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829294.484751:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2300 00000100:00100000:3.0:1787829294.484764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829294.484768:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829294.484776:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829294.484817:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518063616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829294.484841:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829294.484844:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829294.484846:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829294.484849:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2300 00000100:00080000:0.0:1787829294.484851:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829294.484854:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829294.484864:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829294.484875:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518063744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829294.484879:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518063616:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829294.484892:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829294.484905:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829294.484906:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829294.484909:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2300 00000100:00080000:0.0:1787829294.484910:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829294.484913:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829294.484917:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829294.484926:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518063872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829294.484932:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829294.484936:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829294.484938:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829294.484940:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2300 00000100:00080000:0.0:1787829294.484944:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829294.484946:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829294.484949:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829294.484956:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518064000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829294.484962:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829294.484964:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829294.484965:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829294.484968:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2300 00000100:00080000:0.0:1787829294.484971:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829294.484973:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829294.484977:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829294.484982:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829294.484990:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518064128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829294.484997:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829294.485000:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829294.485001:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829294.485003:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2300 00000100:00080000:0.0:1787829294.485006:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829294.485008:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829294.485013:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829294.485049:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff8b2e88040b40 x1874672518063616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829294.485051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518064256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829294.485082:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829294.485085:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829294.485087:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829294.485092:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2300 00000100:00100000:3.0:1787829294.485092:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518063616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829294.485095:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829294.485113:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829294.485120:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829294.485121:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829294.485127:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829294.485130:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518064384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829294.485133:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518063616:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829294.485162:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518063744:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829294.485183:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829294.485192:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff8b2e88041e00 x1874672518063744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829294.485201:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518063744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.485208:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829294.485210:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829294.485212:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518063744:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829294.485216:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518063872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829294.485228:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829294.485234:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff8b2e88040780 x1874672518063872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829294.485242:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518063872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.485252:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829294.485254:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829294.485257:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518063872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829294.485260:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518064000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829294.485269:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829294.485275:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff8b2e88042d00 x1874672518064000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829294.485281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518064000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.485286:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829294.485288:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829294.485290:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518064000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829294.485293:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518064128:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829294.485303:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829294.485309:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff8b2e880421c0 x1874672518064128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829294.485317:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518064128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.485330:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829294.485332:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829294.485334:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518064128:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829294.485337:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518064256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829294.485347:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829294.485352:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff8b2e880412c0 x1874672518064256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829294.485359:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518064256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.485363:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829294.485365:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829294.485367:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518064256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829294.485370:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518064384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829294.485380:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829294.485386:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829294/real 1787829294] req@ffff8b2e88043840 x1874672518064384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829294.485393:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518064384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829319 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829294.485398:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829294.485400:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829294.485402:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518064384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829294.804736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829295.316762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829295.316820:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829295.508765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829295.508765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829295.828771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829296.340744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829296.340823:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829296.532743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829296.532743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829296.852758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829297.364743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829297.364837:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829297.556825:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829297.556831:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829297.876706:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829298.388742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829298.388829:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829298.580736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829298.580767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829298.900800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829299.412738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829299.412892:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829299.604758:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829299.604760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829299.604769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829299.604772:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829299.604781:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829299.604792:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2306 00000100:00080000:0.0:1787829299.604802:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829299.604808:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829299.604842:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518064512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829299.604866:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829299.604868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829299.604870:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829299.604873:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2306 00000100:00080000:0.0:1787829299.604875:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829299.604878:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829299.604887:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829299.604900:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518064512:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829299.604901:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518064640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829299.604914:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829299.604916:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829299.604917:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829299.604920:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2306 00000100:00080000:0.0:1787829299.604922:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829299.604925:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829299.604929:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829299.604938:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518064768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829299.604944:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829299.604947:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829299.604948:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829299.604950:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2306 00000100:00080000:0.0:1787829299.604953:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829299.604955:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829299.604958:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829299.604963:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829299.604967:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518064896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829299.604979:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829299.604995:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829299.604996:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829299.605000:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2306 00000100:00080000:0.0:1787829299.605003:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829299.605006:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829299.605009:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829299.605037:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518065024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829299.605045:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829299.605049:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829299.605050:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829299] req@ffff8b2e88042580 x1874672518064512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829299.605053:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829299.605062:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2306 00000100:00080000:0.0:1787829299.605068:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829299.605074:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829299.605078:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829299.605083:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518064512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829299.605089:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518065152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829299.605106:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829299.605107:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829299.605114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829299.605116:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829299.605117:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829299.605123:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518064512:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829299.605124:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2306 00000100:00080000:0.0:1787829299.605131:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829299.605136:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829299.605139:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829299.605146:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518065280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829299.605168:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518064640:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829299.605195:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829299.605204:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829299] req@ffff8b2e88043840 x1874672518064640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829299.605213:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518064640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829299.605219:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829299.605221:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829299.605223:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518064640:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829299.605227:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518064768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829299.605240:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829299.605246:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829299] req@ffff8b2e880412c0 x1874672518064768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829299.605253:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518064768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829299.605258:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829299.605260:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829299.605262:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518064768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829299.605266:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518064896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829299.605275:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829299.605280:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829299] req@ffff8b2e880421c0 x1874672518064896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829299.605287:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518064896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829299.605292:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829299.605294:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829299.605296:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518064896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829299.605299:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518065024:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829299.605308:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829299.605314:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829299] req@ffff8b2e88042d00 x1874672518065024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829299.605337:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518065024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829299.605343:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829299.605345:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829299.605347:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518065024:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829299.605350:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518065152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829299.605360:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829299.605365:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829299] req@ffff8b2e88040780 x1874672518065152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829299.605372:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518065152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829299.605377:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829299.605378:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829299.605380:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518065152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829299.605400:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518065280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829299.605409:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829299.605415:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829299/real 1787829299] req@ffff8b2e88041e00 x1874672518065280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829299.605422:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518065280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829324 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829299.605431:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829299.605433:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829299.605435:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518065280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829299.924726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829300.436759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829300.436846:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829300.628731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829300.628731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829300.948794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829301.460773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829301.460860:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829301.652766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829301.652766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829301.972766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829302.484751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829302.484837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829302.676718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829302.676725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829302.996715:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829303.508847:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829303.508917:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829303.700806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829303.700828:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829304.020768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829304.532751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829304.532820:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829304.724747:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829304.724748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829304.724749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829304.724760:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829304.724765:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829304.724783:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2311 00000100:00080000:0.0:1787829304.724794:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829304.724802:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829304.724840:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518065408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829304.724867:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.724871:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829304.724873:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829304.724878:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2311 00000100:00080000:0.0:1787829304.724882:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829304.724887:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829304.724896:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518065408:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829304.724897:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829304.724914:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518065536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829304.724927:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.724934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829304.724936:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829304.724940:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2311 00000100:00080000:0.0:1787829304.724959:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829304.724964:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829304.724967:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829304.724972:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829304.724990:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518065664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829304.725002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.725006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829304.725008:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829304.725012:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff8b2e88040b40 x1874672518065408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829304.725016:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2311 00000100:00080000:0.0:1787829304.725024:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829304.725030:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829304.725034:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518065408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829304.725037:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829304.725044:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829304.725048:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829304.725055:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518065408:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829304.725057:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518065792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829304.725069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829304.725073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829304.725075:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829304.725077:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518065536:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829304.725083:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2311 00000100:00080000:0.0:1787829304.725088:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829304.725093:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829304.725099:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829304.725100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829304.725113:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff8b2e88040f00 x1874672518065536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829304.725118:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518065920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829304.725141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829304.725142:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518065536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829304.725150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829304.725156:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829304.725164:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829304.725165:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2311 00000100:00080000:3.0:1787829304.725173:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829304.725174:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829304.725180:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518065536:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829304.725183:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829304.725192:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829304.725192:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518065664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829304.725212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518066048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829304.725224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829304.725224:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829304.725231:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829304.725234:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829304.725241:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff8b2e88041a40 x1874672518065664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829304.725242:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2311 00000100:00080000:0.0:1787829304.725249:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829304.725257:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829304.725264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829304.725265:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518065664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829304.725278:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829304.725280:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518066176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829304.725286:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829304.725295:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518065664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829304.725306:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518065792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829304.725325:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829304.725333:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff8b2e88040000 x1874672518065792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829304.725344:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518065792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829304.725352:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829304.725355:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829304.725359:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518065792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829304.725375:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518065920:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829304.725395:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829304.725407:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff8b2e880430c0 x1874672518065920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829304.725420:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518065920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829304.725429:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829304.725432:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829304.725435:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518065920:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829304.725442:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518066048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829304.725461:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829304.725473:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff8b2e88043480 x1874672518066048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829304.725485:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518066048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829304.725493:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829304.725496:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829304.725499:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518066048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829304.725505:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518066176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829304.725525:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829304.725537:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829304/real 1787829304] req@ffff8b2e88042940 x1874672518066176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829304.725549:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518066176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829329 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829304.725558:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829304.725561:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829304.725564:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518066176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829305.044749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829305.556772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829305.556833:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829305.748748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829305.748748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829306.068735:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829306.580729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829306.580788:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829306.772748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829306.772749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829307.092730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829307.604735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829307.604797:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829307.796744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829307.796803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829308.116759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829308.628744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829308.628846:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829308.820780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829308.820784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829309.140760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829309.652780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829309.652856:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829309.844771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829309.844771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829309.844981:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829309.844988:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829309.844992:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829309.845001:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2316 00000100:00080000:2.0:1787829309.845019:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829309.845027:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829309.845069:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518066304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829309.845094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829309.845097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829309.845098:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829309.845102:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2316 00000100:00080000:2.0:1787829309.845105:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829309.845108:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829309.845117:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829309.845127:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518066432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829309.845135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829309.845141:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829309.845141:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518066304:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829309.845145:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829309.845152:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2316 00000100:00080000:2.0:1787829309.845154:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829309.845156:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829309.845159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829309.845170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518066560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829309.845180:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829309.845184:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829309.845186:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829309.845190:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2316 00000100:00080000:2.0:1787829309.845199:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829309.845212:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829309.845215:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787829309.845222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829309.845237:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518066688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829309.845250:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829309.845253:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829309.845255:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829309.845258:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2316 00000100:00080000:2.0:1787829309.845263:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829309.845263:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff8b2e8801e940 x1874672518066304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829309.845271:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829309.845278:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829309.845287:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518066304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829309.845290:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518066816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829309.845309:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.845317:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829309.845318:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829309.845324:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829309.845325:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518066304:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829309.845328:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829309.845334:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2316 00000100:00080000:2.0:1787829309.845336:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829309.845339:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829309.845342:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829309.845350:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518066432:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787829309.845352:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518066944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829309.845364:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829309.845366:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829309.845368:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829309.845371:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2316 00000100:00080000:2.0:1787829309.845376:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829309.845379:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829309.845385:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787829309.845386:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829309.845399:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518067072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829309.845408:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff8b2e8801e580 x1874672518066432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829309.845422:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518066432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829309.845429:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.845432:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829309.845434:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518066432:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829309.845439:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518066560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829309.845456:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829309.845467:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff8b2e8801c000 x1874672518066560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829309.845479:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518066560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829309.845488:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.845492:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829309.845496:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518066560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829309.845503:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518066688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829309.845522:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829309.845534:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff8b2e8801cf00 x1874672518066688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829309.845547:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518066688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829309.845555:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.845559:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829309.845562:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518066688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829309.845567:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518066816:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829309.845587:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829309.845597:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff8b2e8801f480 x1874672518066816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829309.845609:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518066816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829309.845618:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.845622:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829309.845626:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518066816:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829309.845669:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518066944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829309.845691:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829309.845704:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff8b2e8801fc00 x1874672518066944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829309.845717:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518066944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829309.845726:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.845729:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829309.845734:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518066944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829309.845741:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518067072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829309.845759:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829309.845768:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829309/real 1787829309] req@ffff8b2e8801da40 x1874672518067072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829309.845776:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518067072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829334 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829309.845782:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829309.845784:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829309.845786:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518067072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829310.164748:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787829310.676797:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829310.676848:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829310.868740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829310.868740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829311.188723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829311.700778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829311.700879:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829311.892740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829311.892740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829312.212756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829312.724779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829312.724870:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829312.916733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829312.916788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829313.236758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829313.748820:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829313.748878:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829313.940735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829313.940747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829314.260760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829314.772768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829314.772796:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829314.964721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829314.964734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829314.964789:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.964801:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829314.964807:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.964819:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2321 00000100:00080000:1.0:1787829314.964832:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829314.964843:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829314.964903:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518067200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829314.964940:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.964945:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829314.964949:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.964956:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2321 00000100:00080000:1.0:1787829314.964961:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829314.964967:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829314.964977:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518067200:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787829314.964981:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829314.965005:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518067328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829314.965021:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.965027:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829314.965029:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.965034:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2321 00000100:00080000:1.0:1787829314.965041:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829314.965047:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829314.965050:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787829314.965060:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829314.965079:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518067456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829314.965093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.965097:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829314.965104:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829314.965104:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff8b2e88074b40 x1874672518067200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829314.965119:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2321 00000100:00080000:1.0:1787829314.965130:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829314.965137:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518067200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829314.965139:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829314.965158:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829314.965161:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.965167:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829314.965173:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518067200:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829314.965180:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518067584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829314.965193:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.965198:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829314.965202:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829314.965203:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518067328:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829314.965218:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2321 00000100:00080000:1.0:1787829314.965224:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829314.965233:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829314.965234:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787829314.965245:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829314.965250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff8b2e88075e00 x1874672518067328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829314.965274:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518067712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829314.965275:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518067328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829314.965322:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829314.965326:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829314.965327:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829314.965334:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829314.965335:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829314.965343:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518067328:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829314.965349:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2321 00000100:00080000:1.0:1787829314.965360:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829314.965366:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518067456:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829314.965369:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829314.965380:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829314.965389:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829314.965398:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518067840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829314.965409:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff8b2e880761c0 x1874672518067456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829314.965412:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829314.965429:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829314.965437:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829314.965451:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2321 00000100:00100000:3.0:1787829314.965456:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518067456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829314.965466:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.965476:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829314.965477:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829314.965484:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518067456:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829314.965489:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829314.965496:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518067584:192.168.203.101@tcp:8:kworker.0 00000008:00100000:1.0:1787829314.965498:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829314.965523:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518067968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829314.965536:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829314.965560:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff8b2e88074000 x1874672518067584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829314.965575:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518067584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829314.965581:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.965584:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829314.965586:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518067584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829314.965606:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518067712:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829314.965629:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829314.965642:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff8b2e880770c0 x1874672518067712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829314.965673:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518067712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829314.965680:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.965682:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829314.965684:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518067712:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829314.965689:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518067840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829314.965706:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829314.965713:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff8b2e88076d00 x1874672518067840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829314.965721:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518067840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829314.965726:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.965728:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829314.965731:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518067840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829314.965735:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518067968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829314.965745:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829314.965750:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829314/real 1787829314] req@ffff8b2e88076580 x1874672518067968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829314.965757:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518067968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829339 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829314.965763:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829314.965765:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829314.965766:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518067968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829315.284739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829315.796741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829315.796888:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829315.988714:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829315.988742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829316.308760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829316.820751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829316.820842:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829317.012733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829317.012733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829317.332707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829317.844734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829317.844806:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829318.036724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829318.036732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829318.356719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829318.868754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829318.868834:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829319.060743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829319.060743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829319.380713:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829319.892754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829319.892856:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829320.084713:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829320.084724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829320.084744:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829320.084748:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829320.084751:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829320.084759:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2326 00000100:00080000:2.0:1787829320.084768:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829320.084774:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829320.084808:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518068096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829320.084828:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829320.084831:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829320.084832:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829320.084835:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2326 00000100:00080000:2.0:1787829320.084848:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829320.084851:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829320.084860:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829320.084869:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518068224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829320.084877:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829320.084881:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829320.084882:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518068096:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829320.084883:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829320.084889:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2326 00000100:00080000:2.0:1787829320.084893:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829320.084896:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829320.084899:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829320.084907:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518068352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829320.084913:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829320.084916:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829320.084917:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829320.084919:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2326 00000100:00080000:2.0:1787829320.084922:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829320.084924:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829320.084927:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829320.084934:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518068480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829320.084939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829320.084941:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829320.084943:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829320.084946:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2326 00000100:00080000:2.0:1787829320.084948:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829320.084950:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829320.084953:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829320.084957:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829320.084963:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518068608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829320.084970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829320.084972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829320.084974:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829320.084977:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2326 00000100:00080000:2.0:1787829320.084979:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829320.084982:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829320.084985:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829320.084992:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518068736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829320.084999:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829320.085001:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829320.085002:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829320.085005:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2326 00000100:00080000:2.0:1787829320.085007:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829320.085011:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829320.085013:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829320/real 1787829320] req@ffff8b2e8801c3c0 x1874672518068096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829320.085014:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829320.085025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518068864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829320.085046:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518068096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829320.085060:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829320.085065:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829320.085070:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518068096:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829320.085092:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518068224:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829320.085117:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829320.085132:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829320/real 1787829320] req@ffff8b2e8801cb40 x1874672518068224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829320.085145:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518068224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829320.085153:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829320.085157:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829320.085160:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518068224:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829320.085166:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518068352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829320.085186:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829320.085198:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829320/real 1787829320] req@ffff8b2e8801de00 x1874672518068352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829320.085211:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518068352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829320.085219:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829320.085222:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829320.085225:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518068352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829320.085231:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518068480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829320.085249:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829320.085259:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829320/real 1787829320] req@ffff8b2e8801ed00 x1874672518068480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829320.085271:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518068480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829320.085279:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829320.085282:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829320.085285:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518068480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829320.085291:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518068608:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829320.085309:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829320.085320:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829320/real 1787829320] req@ffff8b2e8801c780 x1874672518068608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829320.085331:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518068608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829320.085340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829320.085344:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829320.085347:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518068608:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829320.085353:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518068736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829320.085373:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829320.085399:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829320/real 1787829320] req@ffff8b2e8801f0c0 x1874672518068736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829320.085412:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518068736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829320.085421:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829320.085423:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829320.085425:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518068736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829320.085429:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518068864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829320.085444:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829320.085452:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829320/real 1787829320] req@ffff8b2e8801d680 x1874672518068864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829320.085459:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518068864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829345 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829320.085464:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829320.085466:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829320.085469:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518068864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829320.404738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829320.916771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829320.916812:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829321.108751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829321.108778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829321.428732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829321.940760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829321.940859:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829322.132705:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829322.132705:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829322.452780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829322.964741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829322.964804:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829323.156737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829323.156737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829323.476755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829323.988711:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829323.988825:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829324.180742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829324.180748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829324.500745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829325.012752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829325.012816:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829325.204739:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829325.204740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829325.204740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829325.204749:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829325.204754:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829325.204764:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2331 00000100:00080000:2.0:1787829325.204773:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829325.204780:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829325.204833:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518068992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829325.204855:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829325.204858:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829325.204859:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829325.204862:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2331 00000100:00080000:2.0:1787829325.204865:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829325.204869:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829325.204877:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829325.204886:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518068992:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829325.204889:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518069120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829325.204900:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829325.204903:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829325.204904:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829325.204907:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2331 00000100:00080000:2.0:1787829325.204909:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829325.204911:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829325.204914:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829325.204923:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518069248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829325.204929:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829325.204931:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829325.204932:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829325.204935:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2331 00000100:00080000:2.0:1787829325.204938:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829325.204940:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829325.204943:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829325.204951:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518069376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829325.204957:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829325.204959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829325.204960:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829325.204963:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2331 00000100:00080000:2.0:1787829325.204965:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829325.204968:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829325.204973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829325.204981:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518069504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829325.204985:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518069120:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829325.204994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829325.205000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829325.205002:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829325.205005:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2331 00000100:00080000:2.0:1787829325.205008:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829325.205011:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829325.205015:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829325.205027:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518069632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829325.205039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829325.205039:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518069248:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787829325.205049:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829325.205052:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829325.205057:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2331 00000100:00080000:2.0:1787829325.205063:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829325.205068:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829325.205072:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518069376:192.168.203.101@tcp:8:kworker.0 00000008:00100000:2.0:1787829325.205077:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829325.205096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518069760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829325.205097:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518069504:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829325.205121:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518069632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829325.205141:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518069760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829325.524729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829326.036747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829326.036850:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829326.228750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829326.228840:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829326.548728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829327.060756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829327.060833:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829327.252881:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829327.252892:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829327.572755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829328.084718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829328.084793:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829328.276765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829328.276942:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:2.0:1787829328.277002:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787829328.277045:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829328.277061:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829328.277101:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829328.277109:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829328.277117:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829328.277125:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829328.277131:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829328.277138:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829328.277209:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829325/real 1787829328] req@ffff8b2e8801c3c0 x1874672518069760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829328.277232:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518069760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829328.277243:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829328.277248:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829328.277253:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518069760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829328.277260:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829325/real 1787829328] req@ffff8b2e8801de00 x1874672518069504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829328.277274:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518069504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829328.277283:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829328.277287:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829328.277291:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518069504:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829328.277298:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829325/real 1787829328] req@ffff8b2e8801cb40 x1874672518069632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829328.277310:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518069632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829328.277317:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829328.277320:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829328.277324:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518069632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829328.277329:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829325/real 1787829328] req@ffff8b2e8801f0c0 x1874672518069120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829328.277341:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518069120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829328.277349:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829328.277352:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829328.277355:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518069120:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829328.277362:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829325/real 1787829328] req@ffff8b2e8801c780 x1874672518069248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829328.277374:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518069248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829328.277382:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829328.277385:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829328.277389:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518069248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829328.277396:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829325/real 1787829328] req@ffff8b2e8801ed00 x1874672518069376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829328.277408:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518069376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829328.277416:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829328.277419:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829328.277422:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518069376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829328.277428:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829325/real 1787829328] req@ffff8b2e8801d680 x1874672518068992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829328.277440:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518068992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829350 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829328.277448:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829328.277452:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829328.277455:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518068992:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829328.596743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829329.108740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829329.108800:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829329.300741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829329.300741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829329.620733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829330.132720:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829330.132803:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829330.324734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829330.324734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829330.324808:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829330.324813:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829330.324816:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829330.324824:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2336 00000100:00080000:0.0:1787829330.324834:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829330.324839:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829330.324867:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518069888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829330.324889:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829330.324891:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829330.324893:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829330.324896:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2336 00000100:00080000:0.0:1787829330.324898:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829330.324901:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829330.324908:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829330.324919:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518070016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829330.324927:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829330.324929:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829330.324930:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829330.324933:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2336 00000100:00080000:0.0:1787829330.324935:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829330.324939:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829330.324941:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518069888:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829330.324945:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829330.324956:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518070144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829330.324974:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829330.324977:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829330.324980:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829330.324984:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2336 00000100:00080000:0.0:1787829330.324991:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829330.324995:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829330.324999:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829330.325002:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829330.325016:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518070272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829330.325033:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829330.325036:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829330.325038:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829330.325048:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2336 00000100:00100000:3.0:1787829330.325051:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff8b2e880403c0 x1874672518069888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829330.325053:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829330.325060:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829330.325069:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829330.325077:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518069888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829330.325082:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518070400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829330.325103:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829330.325111:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829330.325116:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829330.325120:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829330.325124:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2336 00000100:00080000:3.0:1787829330.325129:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829330.325130:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829330.325138:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518069888:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829330.325139:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829330.325149:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829330.325154:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518070016:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829330.325182:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518070528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829330.325187:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829330.325199:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829330.325203:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829330.325204:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829330.325210:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2336 00000100:00100000:3.0:1787829330.325210:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff8b2e88042940 x1874672518070016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829330.325218:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829330.325225:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829330.325233:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829330.325238:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518070016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829330.325248:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518070656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829330.325252:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829330.325262:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829330.325270:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518070016:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829330.325301:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518070144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829330.325327:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829330.325340:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff8b2e88043480 x1874672518070144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829330.325352:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518070144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829330.325360:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829330.325364:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829330.325367:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518070144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829330.325373:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518070272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829330.325393:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829330.325413:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff8b2e880430c0 x1874672518070272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829330.325427:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518070272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829330.325435:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829330.325439:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829330.325442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518070272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829330.325448:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518070400:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829330.325470:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829330.325483:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff8b2e88040000 x1874672518070400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829330.325495:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518070400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829330.325505:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829330.325509:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829330.325512:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518070400:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829330.325519:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518070528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829330.325540:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829330.325732:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff8b2e88041a40 x1874672518070528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829330.325749:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518070528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829330.325756:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829330.325760:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829330.325763:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518070528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829330.325786:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518070656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829330.325810:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829330.325824:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829330/real 1787829330] req@ffff8b2e88040f00 x1874672518070656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829330.325836:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518070656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829355 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829330.325844:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829330.325847:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829330.325850:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518070656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829330.644722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829331.156753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829331.156840:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829331.348711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829331.348757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829331.668706:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829332.180777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829332.180858:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829332.372747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829332.372747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829332.692769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829333.204795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829333.204893:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829333.396800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829333.396800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829333.716720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829334.228778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829334.228851:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829334.420710:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829334.420790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829334.740746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829335.252780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829335.252855:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829335.444777:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829335.444786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829335.444791:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829335.444798:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829335.444801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829335.444812:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2341 00000100:00080000:0.0:1787829335.444826:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829335.444835:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829335.444874:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518070784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829335.444908:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829335.444911:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829335.444912:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829335.444915:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2341 00000100:00080000:0.0:1787829335.444918:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829335.444920:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829335.444928:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829335.444938:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518070912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829335.444946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829335.444949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829335.444952:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829335.444954:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518070784:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829335.444956:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2341 00000100:00080000:0.0:1787829335.444960:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829335.444965:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829335.444969:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829335.444976:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518071040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829335.444983:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829335.444985:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829335.444986:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829335.444989:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2341 00000100:00080000:0.0:1787829335.444992:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829335.444995:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829335.444998:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829335.445005:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518071168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829335.445011:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829335.445013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829335.445015:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829335.445017:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2341 00000100:00080000:0.0:1787829335.445020:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829335.445022:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829335.445025:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829335.445030:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829335.445035:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518071296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829335.445042:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829335.445044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829335.445045:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829335.445048:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2341 00000100:00080000:0.0:1787829335.445050:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829335.445053:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829335.445056:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829335.445062:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518071424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829335.445071:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829335.445072:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829335.445074:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829335.445076:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2341 00000100:00080000:0.0:1787829335.445079:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829335.445081:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829335.445084:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829335.445090:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff8b2e88040b40 x1874672518070784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829335.445093:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518071552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829335.445130:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518070784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829335.445142:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829335.445148:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829335.445152:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518070784:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829335.445174:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518070912:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829335.445200:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829335.445215:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff8b2e88041e00 x1874672518070912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829335.445229:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518070912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829335.445240:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829335.445244:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829335.445247:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518070912:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829335.445254:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518071040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829335.445274:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829335.445283:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff8b2e88040780 x1874672518071040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829335.445292:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518071040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829335.445302:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829335.445304:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829335.445307:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518071040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829335.445311:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518071168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829335.445321:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829335.445327:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff8b2e88042d00 x1874672518071168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829335.445335:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518071168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829335.445340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829335.445342:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829335.445344:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518071168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829335.445347:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518071296:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829335.445360:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829335.445371:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff8b2e880421c0 x1874672518071296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829335.445384:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518071296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829335.445393:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829335.445396:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829335.445400:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518071296:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829335.445407:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518071424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829335.445429:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829335.445443:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff8b2e880412c0 x1874672518071424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829335.445456:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518071424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829335.445464:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829335.445468:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829335.445471:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518071424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829335.445477:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518071552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829335.445499:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829335.445513:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829335/real 1787829335] req@ffff8b2e88043840 x1874672518071552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829335.445527:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518071552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829360 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829335.445535:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829335.445539:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829335.445542:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518071552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829335.764768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829336.276793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829336.276825:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829336.468767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829336.468775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829336.788753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829337.300817:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829337.300882:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829337.492778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829337.492778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829337.812724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829338.324723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829338.324820:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829338.516762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829338.516786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829338.836746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829339.348768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829339.348846:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829339.540747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829339.540747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829339.860763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829340.372731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829340.372809:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829340.564768:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829340.564768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829340.564771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829340.564779:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829340.564784:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.564801:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2346 00000100:00080000:1.0:1787829340.564813:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.564821:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829340.564862:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518071680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829340.564884:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.564887:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829340.564888:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.564892:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2346 00000100:00080000:1.0:1787829340.564895:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.564898:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829340.564908:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829340.564917:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518071680:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829340.564919:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518071808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829340.564931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.564934:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829340.564935:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.564938:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2346 00000100:00080000:1.0:1787829340.564941:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.564944:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829340.564947:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829340.564956:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518071936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829340.564963:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.564966:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829340.564967:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.564970:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2346 00000100:00080000:1.0:1787829340.564974:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.564977:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829340.564982:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829340.564995:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518072064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829340.565006:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829340.565009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.565014:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829340.565016:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.565022:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2346 00000100:00080000:1.0:1787829340.565026:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829340.565030:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829340.565034:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829340.565045:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518072192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829340.565052:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.565055:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829340.565056:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.565060:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2346 00000100:00080000:1.0:1787829340.565063:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829340.565066:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff8b2e88077840 x1874672518071680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829340.565068:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829340.565074:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829340.565087:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518072320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829340.565094:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518071680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829340.565098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829340.565105:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829340.565109:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829340.565115:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2346 00000100:00080000:3.0:1787829340.565116:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829340.565117:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829340.565123:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829340.565124:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829340.565131:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829340.565133:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518071680:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829340.565140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518072448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829340.565160:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518071808:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829340.565181:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829340.565190:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff8b2e88074780 x1874672518071808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.565199:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518071808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.565206:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.565209:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829340.565211:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518071808:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829340.565215:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518071936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829340.565227:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829340.565234:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff8b2e88077c00 x1874672518071936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.565242:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518071936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.565248:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.565250:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829340.565252:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518071936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829340.565256:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518072064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829340.565265:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829340.565271:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff8b2e88075680 x1874672518072064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.565278:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518072064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.565284:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.565286:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829340.565288:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518072064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829340.565291:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518072192:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829340.565301:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829340.565307:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff8b2e88075a40 x1874672518072192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.565313:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518072192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.565320:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.565322:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829340.565324:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518072192:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829340.565327:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518072320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829340.565336:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829340.565342:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff8b2e88077480 x1874672518072320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.565348:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518072320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.565363:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.565368:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829340.565370:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518072320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829340.565375:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518072448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829340.565386:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829340.565393:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829340/real 1787829340] req@ffff8b2e88076940 x1874672518072448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829340.565400:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518072448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829365 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829340.565406:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829340.565407:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829340.565410:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518072448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829340.884761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829341.396750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829341.396861:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829341.588749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829341.588749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829341.908781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829342.420760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829342.420827:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829342.612770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829342.612782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829342.932795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829343.444758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829343.444845:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829343.636762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829343.636767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829343.956767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829344.468771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829344.468821:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829344.660799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829344.660799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829344.980814:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829345.492774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829345.492914:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829345.684751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829345.684758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829345.684783:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829345.684793:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829345.684798:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829345.684807:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2352 00000100:00080000:1.0:1787829345.684818:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829345.684824:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829345.684863:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518072576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829345.684891:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829345.684895:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829345.684897:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829345.684903:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2352 00000100:00080000:1.0:1787829345.684908:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829345.684915:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829345.684926:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829345.684931:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518072576:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829345.684946:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518072704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829345.684957:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829345.684959:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829345.684962:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829345.684965:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2352 00000100:00080000:1.0:1787829345.684968:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829345.684970:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829345.684975:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829345.684987:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518072832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829345.684994:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829345.684997:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829345.684999:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829345.685002:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2352 00000100:00080000:1.0:1787829345.685004:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829345.685006:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829345.685009:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829345.685016:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518072960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829345.685022:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829345.685025:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829345.685034:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829345.685035:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829345.685047:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2352 00000100:00080000:1.0:1787829345.685051:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829345.685054:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829345.685058:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829345.685066:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518073088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829345.685073:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829345.685076:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829345.685078:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829345.685082:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2352 00000100:00080000:1.0:1787829345.685084:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829345.685088:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829345.685094:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff8b2e88074f00 x1874672518072576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829345.685095:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829345.685114:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518073216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829345.685125:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829345.685128:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829345.685129:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829345.685136:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2352 00000100:00100000:3.0:1787829345.685139:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518072576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829345.685140:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829345.685149:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829345.685156:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829345.685170:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518073344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829345.685173:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829345.685187:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829345.685193:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518072576:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829345.685220:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518072704:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829345.685244:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829345.685255:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff8b2e88076940 x1874672518072704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829345.685264:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518072704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829345.685271:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829345.685274:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829345.685277:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518072704:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829345.685281:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518072832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829345.685295:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829345.685303:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff8b2e88077480 x1874672518072832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829345.685311:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518072832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829345.685317:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829345.685319:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829345.685321:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518072832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829345.685325:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518072960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829345.685336:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829345.685346:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff8b2e88075a40 x1874672518072960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829345.685359:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518072960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829345.685368:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829345.685372:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829345.685376:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518072960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829345.685383:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518073088:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829345.685403:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829345.685418:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff8b2e88075680 x1874672518073088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829345.685426:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518073088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829345.685433:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829345.685435:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829345.685438:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518073088:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829345.685442:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518073216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829345.685455:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829345.685461:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff8b2e88077c00 x1874672518073216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829345.685469:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518073216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829345.685475:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829345.685477:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829345.685479:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518073216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829345.685483:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518073344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829345.685493:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829345.685499:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829345/real 1787829345] req@ffff8b2e88074780 x1874672518073344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829345.685510:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518073344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829370 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829345.685518:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829345.685522:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829345.685526:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518073344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829346.004742:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829346.516762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829346.516836:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829346.708749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829346.708749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829347.028755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829347.540721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829347.540836:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829347.732764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829347.732764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829348.052764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829348.564757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829348.564835:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829348.756747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829348.756747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829349.076718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829349.588714:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829349.588795:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829349.780751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829349.780751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829350.100730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829350.612768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829350.612848:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829350.804785:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829350.804795:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829350.804800:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829350.804801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829350.804816:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2357 00000100:00100000:3.0:1787829350.804828:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829350.804830:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829350.804840:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829350.804884:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518073472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829350.804912:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829350.804916:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829350.804918:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829350.804924:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2357 00000100:00080000:0.0:1787829350.804929:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829350.804935:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829350.804944:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518073472:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829350.804946:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829350.804970:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518073600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829350.804984:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829350.804988:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829350.804990:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829350.804994:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2357 00000100:00080000:0.0:1787829350.804999:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829350.805003:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829350.805008:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829350.805014:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829350.805036:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518073728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829350.805048:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff8b2e88042580 x1874672518073472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829350.805049:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829350.805058:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829350.805062:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829350.805072:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2357 00000100:00080000:0.0:1787829350.805077:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829350.805080:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518073472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829350.805085:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829350.805096:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829350.805100:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829350.805106:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829350.805110:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518073472:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829350.805114:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518073856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829350.805137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829350.805141:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518073600:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829350.805144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829350.805148:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829350.805154:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2357 00000100:00080000:0.0:1787829350.805160:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829350.805165:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829350.805168:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829350.805174:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829350.805186:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff8b2e88043840 x1874672518073600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829350.805191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518073984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829350.805214:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829350.805216:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829350.805221:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829350.805221:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518073600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829350.805228:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2357 00000100:00080000:0.0:1787829350.805234:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829350.805240:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829350.805242:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829350.805245:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829350.805250:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829350.805255:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518074112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829350.805257:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518073600:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829350.805273:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829350.805274:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518073728:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829350.805279:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829350.805284:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829350.805290:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2357 00000100:00080000:0.0:1787829350.805292:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829350.805295:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829350.805298:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829350.805307:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518074240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829350.805313:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829350.805332:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff8b2e880412c0 x1874672518073728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829350.805346:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518073728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829350.805355:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829350.805358:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829350.805362:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518073728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829350.805375:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518073856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829350.805392:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829350.805400:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff8b2e880421c0 x1874672518073856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829350.805408:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518073856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829350.805413:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829350.805416:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829350.805418:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518073856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829350.805421:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518073984:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829350.805431:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829350.805437:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff8b2e88042d00 x1874672518073984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829350.805449:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518073984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829350.805455:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829350.805457:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829350.805458:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518073984:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829350.805462:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518074112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829350.805475:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829350.805486:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff8b2e88040780 x1874672518074112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829350.805498:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518074112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829350.805507:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829350.805510:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829350.805514:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518074112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829350.805520:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518074240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829350.805537:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829350.805544:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829350/real 1787829350] req@ffff8b2e88041e00 x1874672518074240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829350.805552:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518074240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829375 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829350.805561:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829350.805563:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829350.805565:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518074240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829351.124750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829351.636772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829351.636830:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829351.828735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829351.828735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829352.148740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829352.660781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829352.660838:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829352.852723:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829352.852725:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829353.172758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829353.684756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829353.684788:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829353.876710:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829353.876725:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829354.196726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829354.708804:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829354.708833:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829354.900745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829354.900746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829355.220754:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829355.732776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829355.732804:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829355.924706:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829355.924729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829355.924781:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.924790:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829355.924795:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829355.924806:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2362 00000100:00080000:1.0:1787829355.924819:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.924827:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829355.924879:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518074368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829355.924902:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.924905:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829355.924906:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829355.924910:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2362 00000100:00080000:1.0:1787829355.924913:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.924917:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829355.924927:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829355.924934:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518074368:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829355.924938:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518074496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829355.924954:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.924958:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829355.924959:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829355.924965:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2362 00000100:00080000:1.0:1787829355.924968:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.924971:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829355.924975:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829355.924984:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518074624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829355.924995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.925000:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829355.925002:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829355.925010:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829355.925016:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2362 00000100:00080000:1.0:1787829355.925023:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.925030:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829355.925038:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829355.925051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518074752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829355.925060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.925063:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829355.925066:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829355.925069:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff8b2e88077840 x1874672518074368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829355.925072:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2362 00000100:00080000:1.0:1787829355.925082:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.925093:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829355.925102:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829355.925113:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518074368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829355.925115:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518074880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829355.925189:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829355.925190:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829355.925196:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829355.925200:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829355.925201:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829355.925216:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2362 00000100:00100000:3.0:1787829355.925218:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518074368:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829355.925220:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829355.925229:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829355.925236:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829355.925247:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518075008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829355.925251:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518074496:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829355.925275:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829355.925278:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829355.925281:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829355.925289:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2362 00000100:00080000:1.0:1787829355.925295:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829355.925298:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829355.925316:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829355.925320:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff8b2e88076580 x1874672518074496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829355.925326:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829355.925349:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518075136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829355.925355:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518074496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829355.925399:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829355.925410:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829355.925417:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518074496:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829355.925427:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518074624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829355.925447:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829355.925456:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff8b2e88076d00 x1874672518074624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829355.925464:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518074624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829355.925469:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829355.925472:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829355.925474:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518074624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829355.925478:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518074752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829355.925488:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829355.925494:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff8b2e880770c0 x1874672518074752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829355.925501:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518074752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829355.925510:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829355.925513:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829355.925515:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518074752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829355.925518:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518074880:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829355.925529:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829355.925535:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff8b2e88074000 x1874672518074880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829355.925542:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518074880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829355.925548:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829355.925549:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829355.925551:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518074880:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829355.925563:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518075008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829355.925574:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829355.925580:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff8b2e880761c0 x1874672518075008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829355.925588:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518075008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829355.925607:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829355.925610:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829355.925613:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518075008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829355.925617:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518075136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829355.925632:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829355.925639:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829355/real 1787829355] req@ffff8b2e88075e00 x1874672518075136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829355.925664:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518075136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829380 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829355.925670:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829355.925672:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829355.925674:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518075136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829356.244726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829356.756776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829356.756895:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829356.948765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829356.948766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829357.268758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829357.780784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829357.780791:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829357.972726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829357.972748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829358.292707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829358.804773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829358.804827:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829358.996730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829358.996753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829359.316752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829359.828737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829359.828811:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829360.020769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829360.020781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829360.340706:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829360.852742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829360.852766:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829361.044719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829361.044745:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829361.044751:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829361.044759:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829361.044767:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2367 00000100:00080000:3.0:1787829361.044776:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829361.044781:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829361.044816:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518075264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829361.044823:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829361.044841:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829361.044844:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829361.044847:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829361.044850:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2367 00000100:00080000:3.0:1787829361.044853:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829361.044856:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829361.044865:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829361.044873:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518075264:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829361.044876:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518075392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829361.044886:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829361.044888:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829361.044889:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829361.044898:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2367 00000100:00080000:3.0:1787829361.044901:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829361.044903:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829361.044907:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829361.044915:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518075520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829361.044921:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829361.044924:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829361.044925:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829361.044927:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2367 00000100:00080000:3.0:1787829361.044929:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829361.044932:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829361.044935:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829361.044942:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518075648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829361.044947:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829361.044965:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829361.044970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829361.044972:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829361.044978:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2367 00000100:00080000:3.0:1787829361.044982:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829361.044987:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829361.044993:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829361.044996:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829361/real 1787829361] req@ffff8b2e8803a1c0 x1874672518075264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829361.045008:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518075776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829361.045019:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518075264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829361.045024:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829361.045032:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829361.045037:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829361.045040:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829361.045044:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2367 00000100:00080000:0.0:1787829361.045048:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829361.045053:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518075264:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829361.045053:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829361.045063:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829361.045068:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829361.045078:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518075904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829361.045085:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518075392:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829361.045088:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829361.045095:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829361.045097:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829361.045100:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2367 00000100:00080000:3.0:1787829361.045103:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829361.045106:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829361.045109:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829361.045117:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518076032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829361.045124:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829361.045143:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829361/real 1787829361] req@ffff8b2e8803b0c0 x1874672518075392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829361.045154:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518075392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829361.045161:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829361.045165:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829361.045169:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518075392:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829361.045175:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518075520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829361.045199:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829361.045211:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829361/real 1787829361] req@ffff8b2e8803b840 x1874672518075520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829361.045221:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518075520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829361.045227:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829361.045229:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829361.045231:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518075520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829361.045235:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518075648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829361.045250:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829361.045256:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829361/real 1787829361] req@ffff8b2e88038f00 x1874672518075648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829361.045271:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518075648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829361.045279:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829361.045282:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829361.045286:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518075648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829361.045292:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518075776:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829361.045312:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829361.045323:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829361/real 1787829361] req@ffff8b2e8803a580 x1874672518075776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829361.045332:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518075776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829361.045337:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829361.045340:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829361.045342:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518075776:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829361.045355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518075904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829361.045368:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829361.045374:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829361/real 1787829361] req@ffff8b2e880383c0 x1874672518075904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829361.045381:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518075904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829361.045391:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829361.045393:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829361.045395:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518075904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829361.045399:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518076032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829361.045408:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829361.045414:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829361/real 1787829361] req@ffff8b2e88039a40 x1874672518076032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829361.045421:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518076032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829386 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829361.045426:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829361.045427:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829361.045429:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518076032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829361.364740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829361.876737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829361.876810:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829362.068761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829362.068761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829362.388769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829362.900754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829362.900823:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829363.092712:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829363.092725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829363.412739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829363.924727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829363.924778:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829364.116741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829364.116742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829364.436719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829364.948733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829364.948807:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829365.140752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829365.140752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829365.460763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829365.972757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829365.972823:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829366.164751:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829366.164753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829366.164764:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829366.164769:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829366.164775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829366.164780:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2372 00000100:00080000:2.0:1787829366.164793:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829366.164802:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829366.164840:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518076160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829366.164861:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829366.164864:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829366.164865:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829366.164868:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2372 00000100:00080000:2.0:1787829366.164871:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829366.164875:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829366.164884:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829366.164893:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518076160:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829366.164895:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518076288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829366.164914:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829366.164918:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829366.164920:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829366.164925:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2372 00000100:00080000:2.0:1787829366.164930:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829366.164934:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829366.164940:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1787829366.164953:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829366.164956:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518076416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829366.164969:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829366.164973:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829366.164976:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829366.164979:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2372 00000100:00080000:2.0:1787829366.164983:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829366.164986:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829366.164991:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829366.164995:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829366/real 1787829366] req@ffff8b2e8801da40 x1874672518076160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829366.165003:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518076544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829366.165012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829366.165014:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518076160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829366.165015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829366.165019:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829366.165027:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2372 00000100:00080000:2.0:1787829366.165031:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829366.165032:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829366.165034:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829366.165039:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829366.165042:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829366.165049:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518076672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829366.165050:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518076160:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829366.165064:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829366.165066:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829366.165068:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829366.165071:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2372 00000100:00080000:2.0:1787829366.165073:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829366.165075:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829366.165078:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829366.165084:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518076288:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787829366.165087:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518076800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829366.165097:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829366.165099:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829366.165100:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829366.165103:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2372 00000100:00080000:2.0:1787829366.165105:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829366.165108:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787829366.165112:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787829366.165112:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829366.165125:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518076928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829366.165129:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829366/real 1787829366] req@ffff8b2e8801fc00 x1874672518076288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829366.165147:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518076288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829366.165153:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829366.165157:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829366.165159:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518076288:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829366.165164:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518076416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829366.165176:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829366.165183:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829366/real 1787829366] req@ffff8b2e8801f480 x1874672518076416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829366.165190:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518076416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829366.165196:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829366.165197:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829366.165199:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518076416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829366.165203:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518076544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829366.165212:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829366.165218:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829366/real 1787829366] req@ffff8b2e8801cf00 x1874672518076544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829366.165225:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518076544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829366.165231:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829366.165233:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829366.165236:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518076544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829366.165242:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518076672:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829366.165260:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829366.165272:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829366/real 1787829366] req@ffff8b2e8801c000 x1874672518076672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829366.165285:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518076672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829366.165293:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829366.165295:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829366.165298:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518076672:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829366.165311:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518076800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829366.165326:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829366.165333:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829366/real 1787829366] req@ffff8b2e8801e580 x1874672518076800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829366.165341:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518076800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829366.165346:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829366.165348:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829366.165350:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518076800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829366.165354:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518076928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829366.165363:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829366.165368:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829366/real 1787829366] req@ffff8b2e8801e940 x1874672518076928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829366.165376:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518076928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829391 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829366.165381:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829366.165382:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829366.165384:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518076928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829366.484737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829366.996729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829366.996752:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829367.188738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829367.188738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829367.508750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829368.020756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829368.020851:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829368.212736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829368.212768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829368.532748:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829369.044716:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829369.044837:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829369.236761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829369.236761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829369.556733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829370.068733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829370.068794:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829370.260753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829370.260753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829370.580730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829371.092741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829371.092794:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829371.284735:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829371.284738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829371.284738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829371.284747:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829371.284751:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829371.284761:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2377 00000100:00080000:2.0:1787829371.284771:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829371.284777:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829371.284812:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518077056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829371.284834:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829371.284837:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829371.284838:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829371.284841:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2377 00000100:00080000:2.0:1787829371.284844:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829371.284847:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829371.284856:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829371.284865:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518077056:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829371.284867:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518077184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829371.284880:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829371.284883:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829371.284885:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829371.284887:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2377 00000100:00080000:2.0:1787829371.284888:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829371.284891:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829371.284894:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829371.284902:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518077312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829371.284908:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829371.284910:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829371.284912:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829371.284914:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2377 00000100:00080000:2.0:1787829371.284916:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829371.284918:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829371.284921:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829371.284927:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518077440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829371.284934:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829371.284937:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829371.284938:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829371.284940:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2377 00000100:00080000:2.0:1787829371.284943:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829371.284946:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829371.284948:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829371.284953:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829371.284959:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518077568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829371.284966:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829371.284969:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829371.284970:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829371.284973:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2377 00000100:00080000:2.0:1787829371.284975:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829371.284977:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829371.284980:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829371.284986:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518077696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829371.284998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829371.284999:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff8b2e8801e940 x1874672518077056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829371.285001:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829371.285004:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829371.285012:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2377 00000100:00080000:2.0:1787829371.285018:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829371.285022:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829371.285026:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829371.285033:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518077056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829371.285035:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518077824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829371.285070:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829371.285076:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829371.285082:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518077056:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829371.285107:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518077184:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829371.285133:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829371.285148:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff8b2e8801e580 x1874672518077184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829371.285163:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518077184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829371.285173:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829371.285176:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829371.285180:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518077184:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829371.285187:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518077312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829371.285210:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829371.285223:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff8b2e8801c000 x1874672518077312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829371.285235:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518077312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829371.285252:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829371.285256:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829371.285259:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518077312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829371.285266:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518077440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829371.285288:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829371.285300:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff8b2e8801cf00 x1874672518077440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829371.285313:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518077440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829371.285322:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829371.285326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829371.285329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518077440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829371.285335:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518077568:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829371.285356:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829371.285368:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff8b2e8801f480 x1874672518077568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829371.285381:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518077568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829371.285390:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829371.285394:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829371.285398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518077568:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829371.285404:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518077696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829371.285422:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829371.285433:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff8b2e8801fc00 x1874672518077696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829371.285444:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518077696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829371.285453:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829371.285456:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829371.285459:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518077696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829371.285465:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518077824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829371.285501:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829371.285514:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829371/real 1787829371] req@ffff8b2e8801da40 x1874672518077824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829371.285526:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518077824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829396 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829371.285534:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829371.285537:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829371.285540:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518077824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829371.604764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829372.116750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829372.116814:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829372.308766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829372.308766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829372.628762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829373.140730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829373.140786:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829373.332720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829373.332728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829373.652732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829374.164766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829374.164820:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829374.356733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829374.356735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829374.676751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829375.188749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829375.188818:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829375.380737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829375.380856:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829375.700710:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829376.212748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829376.212783:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829376.404712:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829376.404738:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829376.404744:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787829376.404745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829376.404748:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829376.404757:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2382 00000100:00080000:0.0:1787829376.404766:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829376.404771:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829376.404805:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518077952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829376.404830:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829376.404835:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829376.404837:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829376.404841:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2382 00000100:00080000:0.0:1787829376.404846:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829376.404851:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829376.404860:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518077952:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829376.404862:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829376.404881:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518078080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829376.404894:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829376.404898:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829376.404900:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829376.404903:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2382 00000100:00080000:0.0:1787829376.404907:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829376.404910:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829376.404915:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829376.404916:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829376.404928:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518078208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829376.404936:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829376.404938:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829376.404940:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829376.404942:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2382 00000100:00080000:0.0:1787829376.404944:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829376.404947:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829376.404951:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829376.404956:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff8b2e88040b40 x1874672518077952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829376.404970:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518078336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829376.404974:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518077952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829376.404992:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829376.404995:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829376.404996:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829376.404998:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829376.405002:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829376.405006:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2382 00000100:00100000:3.0:1787829376.405009:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518077952:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829376.405010:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829376.405017:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829376.405020:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829376.405027:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518078464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829376.405033:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518078080:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829376.405038:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829376.405041:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829376.405043:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829376.405046:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2382 00000100:00080000:0.0:1787829376.405050:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829376.405053:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829376.405058:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829376.405060:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829376.405070:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff8b2e88040f00 x1874672518078080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829376.405073:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518078592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829376.405096:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829376.405098:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518078080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829376.405100:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829376.405104:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829376.405115:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2382 00000100:00080000:3.0:1787829376.405116:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829376.405121:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829376.405121:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829376.405127:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518078080:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829376.405131:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829376.405136:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518078208:192.168.203.101@tcp:8:kworker.0 00000008:00100000:0.0:1787829376.405141:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829376.405153:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829376.405163:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff8b2e88041a40 x1874672518078208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829376.405166:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518078720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829376.405179:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518078208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829376.405189:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829376.405194:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829376.405198:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518078208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829376.405206:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518078336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829376.405226:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829376.405239:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff8b2e88040000 x1874672518078336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829376.405251:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518078336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829376.405257:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829376.405259:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829376.405261:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518078336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829376.405273:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518078464:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829376.405288:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829376.405295:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff8b2e880430c0 x1874672518078464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829376.405303:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518078464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829376.405309:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829376.405311:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829376.405313:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518078464:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829376.405316:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518078592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829376.405325:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829376.405331:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff8b2e88043480 x1874672518078592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829376.405339:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518078592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829376.405344:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829376.405346:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829376.405347:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518078592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829376.405351:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518078720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829376.405360:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829376.405366:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829376/real 1787829376] req@ffff8b2e88042940 x1874672518078720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829376.405373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518078720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829401 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829376.405378:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829376.405380:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829376.405381:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518078720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829376.724717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829377.236726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829377.236774:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829377.428729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829377.428809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829377.748710:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829378.260760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829378.260814:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829378.452751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829378.452751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829378.772784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829379.284764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829379.284789:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829379.476737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829379.476737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829379.796707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829380.308756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829380.308826:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829380.500724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829380.500725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829380.820720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829381.332723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829381.332816:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829381.524744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829381.524744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829381.524759:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829381.524767:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829381.524774:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829381.524787:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2387 00000100:00080000:0.0:1787829381.524803:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829381.524815:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829381.524860:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518078848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829381.524890:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829381.524893:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829381.524895:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829381.524898:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2387 00000100:00080000:0.0:1787829381.524902:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829381.524905:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829381.524917:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829381.524925:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518078848:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829381.524935:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518078976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829381.524949:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829381.524953:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829381.524955:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829381.524961:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2387 00000100:00080000:0.0:1787829381.524965:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829381.524968:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829381.524973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829381.524984:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518079104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829381.524991:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829381.524993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829381.524995:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829381.525000:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829381.525003:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2387 00000100:00080000:0.0:1787829381.525008:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829381.525011:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829381.525015:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829381.525024:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518079232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829381.525030:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829381.525033:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829381.525035:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829381.525037:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2387 00000100:00080000:0.0:1787829381.525040:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829381.525044:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829381.525047:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829381.525051:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff8b2e880403c0 x1874672518078848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829381.525059:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518079360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829381.525069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829381.525072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829381.525075:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829381.525076:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518078848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829381.525080:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2387 00000100:00080000:0.0:1787829381.525086:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829381.525091:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829381.525095:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829381.525096:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829381.525103:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829381.525107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518079488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829381.525110:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518078848:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829381.525129:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829381.525131:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829381.525133:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829381.525136:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2387 00000100:00080000:0.0:1787829381.525139:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829381.525142:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518078976:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829381.525143:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829381.525151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829381.525161:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518079616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829381.525173:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829381.525186:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff8b2e88042940 x1874672518078976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829381.525196:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518078976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829381.525202:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829381.525205:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829381.525207:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518078976:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829381.525212:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518079104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829381.525224:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829381.525231:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff8b2e88043480 x1874672518079104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829381.525239:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518079104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829381.525249:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829381.525251:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829381.525253:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518079104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829381.525257:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518079232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829381.525266:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829381.525272:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff8b2e880430c0 x1874672518079232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829381.525279:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518079232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829381.525284:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829381.525286:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829381.525289:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518079232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829381.525293:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518079360:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829381.525302:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829381.525308:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff8b2e88040000 x1874672518079360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829381.525315:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518079360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829381.525320:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829381.525322:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829381.525325:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518079360:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829381.525329:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518079488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829381.525339:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829381.525344:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff8b2e88041a40 x1874672518079488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829381.525351:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518079488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829381.525365:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829381.525367:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829381.525370:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518079488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829381.525383:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518079616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829381.525398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829381.525405:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829381/real 1787829381] req@ffff8b2e88040f00 x1874672518079616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829381.525427:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518079616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829406 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829381.525433:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829381.525435:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829381.525438:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518079616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829381.844774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829382.356739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829382.356815:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829382.548734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829382.548780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829382.868743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829383.380722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829383.380794:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829383.572746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829383.572747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829383.892738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829384.404737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829384.404836:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829384.596712:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829384.596712:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829384.916766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829385.428755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829385.428865:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829385.620756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829385.620756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829385.940756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829386.452727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829386.452832:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829386.644757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829386.644757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829386.644797:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829386.644808:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829386.644812:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829386.644824:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2393 00000100:00080000:1.0:1787829386.644844:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829386.644852:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829386.644892:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518079744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829386.644926:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829386.644930:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829386.644932:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829386.644937:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2393 00000100:00080000:1.0:1787829386.644942:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829386.644949:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829386.644959:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829386.644970:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518079744:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829386.644981:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518079872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829386.644997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829386.645001:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829386.645003:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829386.645008:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2393 00000100:00080000:1.0:1787829386.645012:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829386.645017:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829386.645023:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829386.645043:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518080000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829386.645055:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829386.645059:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829386.645060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829386.645065:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829386.645070:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2393 00000100:00080000:1.0:1787829386.645075:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829386.645078:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829386.645084:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829386.645093:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518080128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829386.645101:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829386.645103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829386.645105:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829386.645107:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2393 00000100:00080000:1.0:1787829386.645110:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829386.645114:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829386.645118:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829386.645124:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff8b2e88074b40 x1874672518079744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829386.645133:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518080256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829386.645166:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829386.645170:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829386.645172:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829386.645175:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518079744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829386.645182:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2393 00000100:00080000:1.0:1787829386.645194:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829386.645202:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829386.645204:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829386.645208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829386.645215:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829386.645222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518080384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829386.645238:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829386.645244:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518079744:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829386.645246:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829386.645250:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829386.645259:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2393 00000100:00080000:1.0:1787829386.645263:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829386.645269:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829386.645274:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829386.645285:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518079872:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829386.645292:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518080512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829386.645331:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829386.645353:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff8b2e88075e00 x1874672518079872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829386.645369:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518079872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829386.645394:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829386.645400:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829386.645403:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518079872:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829386.645416:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518080000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829386.645442:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829386.645457:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff8b2e880761c0 x1874672518080000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829386.645471:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518080000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829386.645480:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829386.645484:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829386.645488:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518080000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829386.645496:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518080128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829386.645519:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829386.645532:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff8b2e88074000 x1874672518080128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829386.645546:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518080128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829386.645555:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829386.645559:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829386.645563:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518080128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829386.645569:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518080256:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829386.645586:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829386.645595:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff8b2e880770c0 x1874672518080256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829386.645604:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518080256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829386.645610:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829386.645612:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829386.645614:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518080256:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829386.645618:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518080384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829386.645628:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829386.645634:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff8b2e88076d00 x1874672518080384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829386.645642:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518080384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829386.645667:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829386.645669:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829386.645671:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518080384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829386.645688:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518080512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829386.645702:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829386.645710:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829386/real 1787829386] req@ffff8b2e88076580 x1874672518080512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829386.645725:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518080512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829411 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829386.645736:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829386.645738:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829386.645740:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518080512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829386.964787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829387.476776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829387.476860:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829387.668743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829387.668743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829387.988762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829388.500731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829388.500816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829388.692758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829388.692780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829389.012764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829389.524740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829389.524807:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829389.716765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829389.716765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829390.036724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829390.548712:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829390.548801:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829390.740710:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829390.740727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829391.060749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829391.572770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829391.572803:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829391.764748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829391.764748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829391.764773:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829391.764779:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829391.764784:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829391.764795:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2398 00000100:00080000:0.0:1787829391.764807:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829391.764815:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829391.764856:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518080640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829391.764878:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829391.764881:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829391.764882:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829391.764886:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2398 00000100:00080000:0.0:1787829391.764888:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829391.764891:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829391.764903:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829391.764913:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518080640:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829391.764915:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518080768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829391.764927:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829391.764930:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829391.764931:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829391.764933:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2398 00000100:00080000:0.0:1787829391.764935:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829391.764938:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829391.764941:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829391.764949:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518080896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829391.764956:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829391.764959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829391.764960:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829391.764962:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2398 00000100:00080000:0.0:1787829391.764965:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829391.764967:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829391.764970:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829391.764976:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518081024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829391.764982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829391.764985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829391.764986:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829391.764990:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2398 00000100:00080000:0.0:1787829391.764993:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829391.764996:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829391.765000:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829391.765008:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518081152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829391.765014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829391.765017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829391.765018:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829391.765022:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2398 00000100:00080000:0.0:1787829391.765025:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829391.765028:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829391.765030:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829391.765042:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518081280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829391.765042:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518080768:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829391.765057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829391.765059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829391.765061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829391.765064:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2398 00000100:00080000:0.0:1787829391.765067:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829391.765070:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829391.765073:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829391.765083:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518081408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829391.765091:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518080896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829391.765115:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518081024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829391.765129:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518081152:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829391.765154:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518081280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829391.765180:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518081408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829392.084759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829392.596731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829392.596806:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829392.788713:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829392.788731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829393.108737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829393.620757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829393.620822:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829393.812763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829393.812763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829394.132781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829394.644769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829394.644795:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829394.836717:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829394.836741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:3.0:1787829394.836995:0:2336:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:3.0:1787829394.837035:0:2336:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829394.837052:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829394.837094:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829394.837115:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829394.837129:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829394.837149:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787829394.837166:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829394] req@ffff8b2e88041e00 x1874672518080768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:3.0:1787829394.837166:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829394.837194:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787829394.837223:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518080768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829394.837241:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829394.837248:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829394.837253:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518080768:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829394.837263:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829394] req@ffff8b2e88040780 x1874672518080896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829394.837281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518080896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829394.837289:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829394.837293:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829394.837296:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518080896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829394.837304:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829394] req@ffff8b2e88042d00 x1874672518081024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829394.837316:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518081024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829394.837339:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829394.837344:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829394.837348:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518081024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829394.837367:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829394] req@ffff8b2e880421c0 x1874672518081152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829394.837381:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518081152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829394.837390:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829394.837394:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829394.837398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518081152:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829394.837407:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829394] req@ffff8b2e88040b40 x1874672518080640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829394.837422:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518080640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829394.837431:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829394.837435:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829394.837439:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518080640:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829394.837480:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829394] req@ffff8b2e880412c0 x1874672518081280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829394.837494:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518081280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829394.837502:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829394.837506:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829394.837509:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518081280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829394.837517:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829391/real 1787829394] req@ffff8b2e88043840 x1874672518081408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829394.837528:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518081408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829416 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829394.837536:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829394.837540:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829394.837543:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518081408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829395.156724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829395.668782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829395.668823:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829395.860726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829395.860728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829396.180802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829396.692780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829396.692811:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829396.884711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829396.884801:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829396.884812:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829396.884819:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829396.884827:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829396.884833:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2403 00000100:00080000:2.0:1787829396.884850:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829396.884861:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829396.884911:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518081536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829396.884960:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829396.884965:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829396.884967:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829396.884972:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2403 00000100:00080000:2.0:1787829396.884977:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829396.884982:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829396.884994:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829396.885009:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518081664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829396.885028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829396.885033:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829396.885035:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829396.885039:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2403 00000100:00080000:2.0:1787829396.885043:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829396.885047:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829396.885053:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829396.885066:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518081792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829396.885077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829396.885081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829396.885083:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829396.885087:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2403 00000100:00080000:2.0:1787829396.885090:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829396.885095:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829396.885100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829396.885112:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518081920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829396.885123:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829396.885127:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829396.885129:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829396.885135:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2403 00000100:00080000:2.0:1787829396.885139:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829396.885144:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829396.885149:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829396.885161:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518082048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829396.885171:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829396.885174:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829396.885176:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829396.885181:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2403 00000100:00080000:2.0:1787829396.885185:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829396.885189:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829396.885194:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829396.885207:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518082176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829396.885217:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829396.885220:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829396.885222:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829396.885226:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2403 00000100:00080000:2.0:1787829396.885230:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829396.885234:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829396.885240:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829396.885251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518082304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829396.885374:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518081536:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787829396.885464:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829396.885524:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff8b2e8801d680 x1874672518081536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.885544:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518081536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.885557:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.885561:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829396.885566:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518081536:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829396.885574:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518081664:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829396.885600:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829396.885616:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff8b2e8801ed00 x1874672518081664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.885630:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518081664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.885640:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.885643:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829396.885677:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518081664:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829396.885685:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518081792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829396.885710:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829396.885723:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff8b2e8801c780 x1874672518081792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.885737:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518081792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.885746:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.885750:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829396.885754:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518081792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829396.885761:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518081920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829396.885782:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829396.885794:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff8b2e8801f0c0 x1874672518081920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.885807:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518081920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.885816:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.885820:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829396.885824:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518081920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829396.885830:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518082048:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829396.885851:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829396.885863:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff8b2e8801cb40 x1874672518082048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.885877:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518082048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.885887:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.885890:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829396.885894:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518082048:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829396.885901:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518082176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829396.885923:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829396.885938:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff8b2e8801de00 x1874672518082176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.885952:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518082176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.885961:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.885967:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829396.885972:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518082176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829396.885982:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518082304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829396.886008:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829396.886022:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829396/real 1787829396] req@ffff8b2e8801c3c0 x1874672518082304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829396.886035:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518082304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829421 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829396.886044:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829396.886047:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829396.886050:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518082304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829397.204707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829397.716756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829397.716813:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829397.908735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829397.908735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829398.228744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829398.740732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829398.740758:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829398.932792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829398.932792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829399.252700:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829399.764756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829399.764868:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829399.956726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829399.956726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829400.276745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829400.788724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829400.788776:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829400.980781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829400.980781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829401.300759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829401.812763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829401.812807:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829402.004749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829402.004749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829402.004770:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829402.004780:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829402.004785:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829402.004796:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2408 00000100:00080000:1.0:1787829402.004807:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829402.004814:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829402.004853:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518082432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829402.004886:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829402.004890:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829402.004893:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829402.004898:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2408 00000100:00080000:1.0:1787829402.004903:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829402.004907:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829402.004917:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829402.004932:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518082560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829402.004936:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518082432:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829402.004953:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829402.004957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829402.004959:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829402.004963:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2408 00000100:00080000:1.0:1787829402.004967:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829402.004970:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829402.004976:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829402.004987:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518082688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829402.004997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829402.005000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829402.005002:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829402.005008:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829402.005010:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2408 00000100:00080000:1.0:1787829402.005015:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829402.005018:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829402.005023:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829402.005034:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518082816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829402.005045:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829402.005049:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829402.005051:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829402.005056:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2408 00000100:00100000:0.0:1787829402.005061:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829402/real 1787829402] req@ffff8b2e88077840 x1874672518082432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829402.005062:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829402.005072:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829402.005080:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829402.005084:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518082432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829402.005094:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829402.005096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518082944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829402.005099:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829402.005107:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518082432:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829402.005118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829402.005122:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829402.005124:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829402.005130:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2408 00000100:00080000:1.0:1787829402.005134:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829402.005139:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518082560:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829402.005140:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829402.005149:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829402.005164:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518083072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829402.005166:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829402.005180:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829402/real 1787829402] req@ffff8b2e88074780 x1874672518082560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829402.005181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829402.005192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829402.005195:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829402.005198:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518082560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829402.005203:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2408 00000100:00080000:0.0:1787829402.005210:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829402.005211:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829402.005215:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829402.005221:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518082560:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829402.005221:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829402.005229:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518082688:192.168.203.101@tcp:8:kworker.0 00000008:00100000:1.0:1787829402.005231:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829402.005249:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518083200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829402.005252:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829402.005263:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829402/real 1787829402] req@ffff8b2e88077c00 x1874672518082688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829402.005289:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518082688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829402.005295:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829402.005298:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829402.005300:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518082688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829402.005305:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518082816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829402.005319:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829402.005327:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829402/real 1787829402] req@ffff8b2e88075680 x1874672518082816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829402.005334:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518082816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829402.005339:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829402.005341:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829402.005343:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518082816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829402.005347:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518082944:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829402.005357:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829402.005363:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829402/real 1787829402] req@ffff8b2e88075a40 x1874672518082944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829402.005370:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518082944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829402.005375:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829402.005377:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829402.005379:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518082944:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829402.005391:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518083072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829402.005408:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829402.005415:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829402/real 1787829402] req@ffff8b2e88077480 x1874672518083072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829402.005423:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518083072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829402.005433:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829402.005436:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829402.005439:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518083072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829402.005445:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518083200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829402.005466:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829402.005477:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829402/real 1787829402] req@ffff8b2e88076940 x1874672518083200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829402.005486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518083200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829427 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829402.005492:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829402.005494:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829402.005496:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518083200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829402.324763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829402.836753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829402.836824:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829403.028756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829403.028756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829403.348732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829403.860740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829403.860770:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829404.052761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829404.052761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829404.372718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829404.884798:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829404.884875:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829405.076764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829405.076786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829405.396800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829405.908733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829405.908843:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829406.100755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829406.100766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829406.420757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829406.932800:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829406.932837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829407.124786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829407.124787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829407.124797:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829407.124808:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829407.124814:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829407.124829:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2413 00000100:00080000:1.0:1787829407.124846:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829407.124854:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829407.124896:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518083328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829407.124920:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829407.124924:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829407.124925:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829407.124931:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2413 00000100:00080000:1.0:1787829407.124934:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829407.124937:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829407.124948:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829407.124956:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518083328:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829407.124962:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518083456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829407.124972:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829407.124975:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829407.124976:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829407.124979:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2413 00000100:00080000:1.0:1787829407.124983:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829407.124986:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829407.124990:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829407.125001:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518083584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829407.125008:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829407.125011:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829407.125012:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829407.125015:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2413 00000100:00080000:1.0:1787829407.125018:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829407.125026:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787829407.125028:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787829407.125031:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829407.125043:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518083712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829407.125051:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829407.125055:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829407.125057:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829407.125063:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2413 00000100:00080000:1.0:1787829407.125068:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829407.125072:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829407.125078:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829407/real 1787829407] req@ffff8b2e88074f00 x1874672518083328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829407.125081:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829407.125095:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518083840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829407.125131:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829407.125134:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829407.125135:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829407.125141:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518083328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829407.125142:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2413 00000100:00080000:1.0:1787829407.125148:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829407.125153:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829407.125160:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829407.125177:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518083968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829407.125179:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829407.125194:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829407.125199:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829407.125201:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829407.125206:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829407.125212:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518083328:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829407.125221:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2413 00000100:00080000:1.0:1787829407.125225:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829407.125231:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829407.125250:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829407.125257:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518083456:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829407.125264:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518084096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829407.125286:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829407.125306:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829407/real 1787829407] req@ffff8b2e88076940 x1874672518083456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829407.125321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518083456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829407.125331:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829407.125335:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829407.125338:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518083456:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829407.125346:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518083584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829407.125363:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829407.125371:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829407/real 1787829407] req@ffff8b2e88077480 x1874672518083584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829407.125379:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518083584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829407.125385:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829407.125387:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829407.125389:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518083584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829407.125394:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518083712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829407.125404:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829407.125410:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829407/real 1787829407] req@ffff8b2e88075a40 x1874672518083712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829407.125417:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518083712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829407.125422:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829407.125424:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829407.125426:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518083712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829407.125430:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518083840:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829407.125440:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829407.125445:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829407/real 1787829407] req@ffff8b2e88075680 x1874672518083840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829407.125452:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518083840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829407.125458:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829407.125460:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829407.125461:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518083840:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829407.125465:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518083968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829407.125474:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829407.125480:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829407/real 1787829407] req@ffff8b2e88077c00 x1874672518083968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829407.125488:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518083968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829407.125493:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829407.125494:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829407.125496:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518083968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829407.125509:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518084096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829407.125520:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829407.125526:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829407/real 1787829407] req@ffff8b2e88074780 x1874672518084096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829407.125534:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518084096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829432 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829407.125539:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829407.125541:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829407.125542:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518084096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829407.444721:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829407.956722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829407.956760:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829408.148727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829408.148739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829408.468736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829408.980779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829408.980853:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829409.172754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829409.172776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829409.492726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829410.004761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829410.004787:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829410.196748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829410.196775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829410.516790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829411.028761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829411.028827:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829411.220771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829411.220771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829411.540738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829412.052749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829412.052814:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829412.244731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829412.244736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829412.244748:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829412.244757:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829412.244764:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829412.244777:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2418 00000100:00080000:1.0:1787829412.244792:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829412.244802:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829412.244850:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518084224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829412.244873:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829412.244876:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829412.244878:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829412.244884:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2418 00000100:00080000:1.0:1787829412.244888:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829412.244892:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829412.244903:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829412.244907:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518084224:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829412.244917:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518084352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829412.244925:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829412.244928:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829412.244929:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829412.244932:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2418 00000100:00080000:1.0:1787829412.244934:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829412.244937:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829412.244941:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829412.244949:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518084480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829412.244955:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829412.244957:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829412.244958:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829412.244960:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2418 00000100:00080000:1.0:1787829412.244969:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829412.244972:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829412.244976:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829412.244979:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829412.244988:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518084608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829412.244997:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829412.244999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829412.245001:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829412.245004:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2418 00000100:00080000:1.0:1787829412.245006:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829412.245009:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829412.245012:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829412.245024:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff8b2e88077840 x1874672518084224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829412.245033:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518084736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829412.245042:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829412.245046:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829412.245047:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518084224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829412.245048:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829412.245056:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2418 00000100:00080000:1.0:1787829412.245061:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829412.245067:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829412.245070:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829412.245071:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829412.245077:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829412.245084:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518084864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829412.245085:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518084224:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829412.245098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829412.245100:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829412.245102:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829412.245105:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2418 00000100:00080000:1.0:1787829412.245108:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829412.245110:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829412.245114:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829412.245115:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518084352:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829412.245124:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518084992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829412.245139:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829412.245153:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff8b2e88076580 x1874672518084352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829412.245163:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518084352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829412.245173:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829412.245176:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829412.245180:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518084352:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829412.245187:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518084480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829412.245225:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829412.245239:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff8b2e88076d00 x1874672518084480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829412.245252:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518084480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829412.245268:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829412.245272:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829412.245276:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518084480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829412.245283:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518084608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829412.245302:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829412.245311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff8b2e880770c0 x1874672518084608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829412.245319:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518084608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829412.245325:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829412.245327:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829412.245329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518084608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829412.245333:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518084736:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829412.245344:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829412.245350:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff8b2e88074000 x1874672518084736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829412.245357:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518084736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829412.245363:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829412.245365:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829412.245368:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518084736:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829412.245371:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518084864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829412.245382:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829412.245390:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff8b2e880761c0 x1874672518084864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829412.245402:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518084864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829412.245411:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829412.245414:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829412.245418:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518084864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829412.245437:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518084992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829412.245459:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829412.245472:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829412/real 1787829412] req@ffff8b2e88075e00 x1874672518084992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829412.245484:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518084992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829437 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829412.245493:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829412.245497:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829412.245500:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518084992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829412.564741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829413.076764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829413.076813:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829413.268771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829413.268772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829413.588769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829414.100764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829414.100794:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829414.292741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829414.292742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829414.612777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829415.124750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829415.124821:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829415.316749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829415.316857:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829415.636769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829416.148725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829416.148766:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829416.340760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829416.340760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829416.660726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829417.172762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829417.172823:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829417.364741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829417.364741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829417.364747:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829417.364754:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829417.364759:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829417.364770:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2423 00000100:00080000:1.0:1787829417.364783:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829417.364791:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829417.364833:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518085120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829417.364854:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829417.364857:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829417.364858:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829417.364864:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2423 00000100:00080000:1.0:1787829417.364867:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829417.364870:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829417.364880:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829417.364892:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518085248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829417.364895:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518085120:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829417.364904:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829417.364907:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829417.364909:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829417.364912:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2423 00000100:00080000:1.0:1787829417.364914:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829417.364916:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829417.364919:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829417.364927:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518085376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829417.364932:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829417.364935:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829417.364936:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829417.364939:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2423 00000100:00080000:1.0:1787829417.364942:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829417.364944:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829417.364947:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829417.364954:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518085504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829417.364960:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829417.364963:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829417.364965:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829417.364967:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2423 00000100:00080000:1.0:1787829417.364969:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829417.364971:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829417.364975:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829417.364980:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829417.364985:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518085632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829417.364992:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829417.364995:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829417.364997:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829417.364999:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2423 00000100:00080000:1.0:1787829417.365002:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829417.365004:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829417.365006:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829417.365013:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518085760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829417.365019:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829417.365021:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829417.365022:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829417.365025:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2423 00000100:00080000:1.0:1787829417.365028:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829417.365033:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff8b2e88074b40 x1874672518085120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829417.365033:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829417.365038:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829417.365048:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518085888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829417.365063:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518085120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829417.365073:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829417.365077:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829417.365080:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518085120:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829417.365101:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518085248:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829417.365121:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829417.365131:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff8b2e88075e00 x1874672518085248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829417.365140:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518085248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829417.365146:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829417.365149:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829417.365151:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518085248:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829417.365155:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518085376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829417.365187:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829417.365195:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff8b2e880761c0 x1874672518085376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829417.365204:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518085376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829417.365210:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829417.365212:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829417.365214:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518085376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829417.365218:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518085504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829417.365229:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829417.365235:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff8b2e88074000 x1874672518085504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829417.365242:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518085504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829417.365247:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829417.365249:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829417.365251:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518085504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829417.365255:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518085632:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829417.365271:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829417.365282:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff8b2e880770c0 x1874672518085632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829417.365303:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518085632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829417.365313:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829417.365316:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829417.365319:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518085632:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829417.365323:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518085760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829417.365339:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829417.365346:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff8b2e88076d00 x1874672518085760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829417.365354:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518085760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829417.365363:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829417.365366:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829417.365370:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518085760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829417.365377:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518085888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829417.365398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829417.365409:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829417/real 1787829417] req@ffff8b2e88076580 x1874672518085888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829417.365419:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518085888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829442 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829417.365429:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829417.365432:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829417.365435:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518085888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829417.684763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829418.196731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829418.196797:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829418.388754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829418.388754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829418.708742:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829419.220762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829419.220837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829419.412718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829419.412725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829419.732765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829420.244749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829420.244842:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829420.436766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829420.436771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829420.756779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829421.268771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829421.268861:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829421.460716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829421.460726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829421.780774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829422.292757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829422.292861:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829422.484800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829422.484800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829422.484811:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829422.484823:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829422.484829:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829422.484842:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2428 00000100:00080000:1.0:1787829422.484859:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829422.484869:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829422.484919:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518086016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829422.484950:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829422.484954:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829422.484957:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829422.484965:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2428 00000100:00080000:1.0:1787829422.484970:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829422.484976:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829422.484988:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829422.484995:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518086016:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829422.485007:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518086144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829422.485020:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829422.485025:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829422.485027:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829422.485031:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2428 00000100:00080000:1.0:1787829422.485036:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829422.485040:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829422.485046:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829422.485060:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518086272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829422.485071:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829422.485075:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829422.485079:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829422.485080:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829422.485088:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2428 00000100:00080000:1.0:1787829422.485094:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829422.485099:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829422.485105:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829422.485120:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518086400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829422.485145:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829422.485150:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829422.485152:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829422.485157:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff8b2e88077840 x1874672518086016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829422.485161:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2428 00000100:00080000:1.0:1787829422.485169:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829422.485177:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829422.485183:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829422.485193:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518086016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829422.485200:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518086528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829422.485212:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829422.485217:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829422.485221:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829422.485225:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829422.485230:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518086016:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829422.485231:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829422.485241:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2428 00000100:00080000:1.0:1787829422.485246:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829422.485251:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829422.485257:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829422.485264:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518086144:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829422.485272:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518086656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829422.485285:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829422.485289:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829422.485292:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829422.485299:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829422.485299:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2428 00000100:00080000:1.0:1787829422.485308:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829422.485313:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829422.485319:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829422.485325:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff8b2e88074780 x1874672518086144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829422.485336:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518086784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829422.485345:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518086144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829422.485362:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829422.485367:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829422.485371:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518086144:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829422.485380:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518086272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829422.485404:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829422.485417:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff8b2e88077c00 x1874672518086272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829422.485426:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518086272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829422.485439:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829422.485442:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829422.485445:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518086272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829422.485450:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518086400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829422.485465:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829422.485472:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff8b2e88075680 x1874672518086400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829422.485480:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518086400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829422.485486:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829422.485488:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829422.485491:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518086400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829422.485495:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518086528:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829422.485506:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829422.485512:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff8b2e88075a40 x1874672518086528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829422.485519:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518086528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829422.485525:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829422.485527:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829422.485529:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518086528:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829422.485541:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518086656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829422.485552:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829422.485559:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff8b2e88077480 x1874672518086656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829422.485566:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518086656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829422.485572:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829422.485574:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829422.485575:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518086656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829422.485579:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518086784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829422.485588:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829422.485595:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829422/real 1787829422] req@ffff8b2e88076940 x1874672518086784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829422.485603:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518086784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829447 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829422.485608:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829422.485610:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829422.485611:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518086784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829422.804786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829423.316780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829423.316840:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829423.508716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829423.508748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829423.828718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829424.340773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829424.340850:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829424.532753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829424.532760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829424.852777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829425.364751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829425.364856:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829425.556756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829425.556756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829425.876768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829426.388771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829426.388885:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829426.580779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829426.580817:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829426.900742:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829427.412744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829427.412809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829427.604742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829427.604748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829427.604775:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829427.604784:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829427.604789:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829427.604802:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2434 00000100:00080000:3.0:1787829427.604817:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829427.604826:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829427.604873:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518086912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829427.604899:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829427.604903:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829427.604904:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829427.604908:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2434 00000100:00080000:3.0:1787829427.604911:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829427.604915:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829427.604926:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829427.604941:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518086912:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829427.604941:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518087040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829427.604956:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829427.604959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829427.604961:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829427.604964:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2434 00000100:00080000:3.0:1787829427.604966:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829427.604969:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829427.604973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829427.604981:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518087168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829427.604987:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829427.604990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829427.604991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829427.604994:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2434 00000100:00080000:3.0:1787829427.604996:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829427.604998:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829427.605002:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829427.605008:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518087296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829427.605014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829427.605016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829427.605017:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829427.605020:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2434 00000100:00080000:3.0:1787829427.605023:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829427.605025:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829427.605029:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829427.605036:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829427.605043:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518087424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829427.605055:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829427.605059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829427.605061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829427.605066:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2434 00000100:00080000:3.0:1787829427.605070:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829427.605075:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829427.605081:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829427.605088:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff8b2e880392c0 x1874672518086912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829427.605110:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518087552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829427.605124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829427.605125:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518086912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829427.605130:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829427.605134:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829427.605144:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2434 00000100:00080000:0.0:1787829427.605145:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829427.605148:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829427.605152:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829427.605156:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829427.605159:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518086912:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787829427.605164:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829427.605183:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518087040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829427.605191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518087680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829427.605208:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829427.605222:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff8b2e88039680 x1874672518087040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829427.605231:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518087040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829427.605238:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829427.605240:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829427.605243:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518087040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829427.605247:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518087168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829427.605259:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829427.605264:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff8b2e8803b480 x1874672518087168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829427.605272:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518087168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829427.605277:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829427.605279:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829427.605281:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518087168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829427.605284:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518087296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829427.605294:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829427.605299:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff8b2e88039e00 x1874672518087296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829427.605306:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518087296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829427.605312:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829427.605315:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829427.605317:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518087296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829427.605320:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518087424:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829427.605330:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829427.605335:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff8b2e88039a40 x1874672518087424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829427.605342:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518087424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829427.605348:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829427.605350:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829427.605351:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518087424:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829427.605356:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518087552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829427.605374:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829427.605386:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff8b2e880383c0 x1874672518087552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829427.605399:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518087552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829427.605409:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829427.605412:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829427.605417:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518087552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829427.605432:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518087680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829427.605447:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829427.605455:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829427/real 1787829427] req@ffff8b2e8803a580 x1874672518087680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829427.605463:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518087680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829452 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829427.605469:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829427.605471:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829427.605473:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518087680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829427.924746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829428.436757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829428.436869:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829428.628764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829428.628774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829428.948726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829429.460768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829429.460788:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829429.652735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829429.652740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829429.972753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829430.484762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829430.484849:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829430.676755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829430.676763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829430.996722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829431.508717:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829431.508815:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829431.700732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829431.700732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829432.020747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829432.532730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829432.532828:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829432.724759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829432.724759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829432.724760:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829432.724772:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829432.724777:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829432.724791:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2439 00000100:00080000:3.0:1787829432.724806:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829432.724817:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829432.724866:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518087808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829432.724898:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829432.724903:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829432.724905:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829432.724911:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2439 00000100:00080000:3.0:1787829432.724916:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829432.724922:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829432.724935:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829432.724939:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518087808:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829432.724954:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518087936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829432.724969:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829432.724973:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829432.724976:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829432.724980:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2439 00000100:00080000:3.0:1787829432.724986:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829432.724991:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829432.724996:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829432.725012:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518088064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829432.725014:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829432.725036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829432.725040:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829432.725043:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829432.725049:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2439 00000100:00080000:3.0:1787829432.725067:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829432.725073:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829432.725078:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829432.725085:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829432] req@ffff8b2e88038f00 x1874672518087808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829432.725095:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518088192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829432.725105:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518087808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829432.725110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829432.725119:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829432.725123:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829432.725125:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829432.725130:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2439 00000100:00080000:0.0:1787829432.725131:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829432.725139:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829432.725140:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518087808:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829432.725149:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829432.725155:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829432.725166:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518087936:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829432.725171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518088320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829432.725187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829432.725192:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829432.725194:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829432.725199:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829432.725205:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2439 00000100:00100000:0.0:1787829432.725208:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829432] req@ffff8b2e8803b840 x1874672518087936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829432.725213:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829432.725222:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518087936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829432.725224:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829432.725234:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829432.725236:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829432.725243:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829432.725245:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518087936:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829432.725252:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518088064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829432.725256:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518088448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829432.725269:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829432.725270:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829432.725276:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829432.725280:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829432.725281:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829432] req@ffff8b2e8803b0c0 x1874672518088064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829432.725289:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2439 00000100:00080000:3.0:1787829432.725297:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829432.725299:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518088064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829432.725304:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829432.725314:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829432.725315:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829432.725328:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518088576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829432.725330:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829432.725334:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518088064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829432.725339:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518088192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829432.725366:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829432.725377:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829432] req@ffff8b2e8803a1c0 x1874672518088192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829432.725385:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518088192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829432.725391:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829432.725393:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829432.725396:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518088192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829432.725406:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518088320:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829432.725418:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829432.725424:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829432] req@ffff8b2e88038f00 x1874672518088320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829432.725432:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518088320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829432.725438:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829432.725440:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829432.725442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518088320:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829432.725446:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518088448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829432.725455:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829432.725460:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829432] req@ffff8b2e8803a580 x1874672518088448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829432.725468:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518088448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829432.725473:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829432.725476:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829432.725478:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518088448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829432.725481:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518088576:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829432.725490:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829432.725495:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829432/real 1787829432] req@ffff8b2e880383c0 x1874672518088576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829432.725502:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518088576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829457 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829432.725508:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829432.725509:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829432.725511:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518088576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829433.044787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829433.556726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829433.556825:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829433.748759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829433.748759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829434.068751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829434.580756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829434.580844:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829434.772767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829434.772767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829435.092759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829435.604746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829435.604805:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829435.796770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829435.796770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829436.116729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829436.628762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829436.628869:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829436.820766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829436.820766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829437.140787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829437.652744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829437.652824:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829437.844774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829437.844789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829437.844792:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829437.844801:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829437.844816:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829437.844829:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2444 00000100:00080000:0.0:1787829437.844844:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829437.844854:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829437.844903:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518088704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829437.844935:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829437.844939:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829437.844942:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829437.844947:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2444 00000100:00080000:0.0:1787829437.844954:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829437.844959:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829437.844971:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829437.844973:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518088704:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829437.844993:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518088832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829437.845008:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829437.845013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829437.845015:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829437.845035:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2444 00000100:00080000:0.0:1787829437.845039:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829437.845044:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829437.845051:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829437.845058:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829437.845068:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518088960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829437.845083:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829437.845087:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829437.845089:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829437.845096:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2444 00000100:00080000:0.0:1787829437.845102:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829437.845104:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff8b2e88043840 x1874672518088704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829437.845111:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829437.845129:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829437.845138:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518088704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829437.845148:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518089088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829437.845151:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.845164:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829437.845176:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518088704:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829437.845190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829437.845197:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829437.845200:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829437.845207:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2444 00000100:00100000:3.0:1787829437.845211:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518088832:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829437.845214:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829437.845225:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829437.845233:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829437.845238:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829437.845254:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518089216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829437.845254:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff8b2e880412c0 x1874672518088832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829437.845334:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829437.845336:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518088832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829437.845344:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829437.845348:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829437.845360:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2444 00000100:00080000:3.0:1787829437.845365:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829437.845368:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829437.845375:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829437.845381:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829437.845388:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829437.845398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518088832:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829437.845399:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518089344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829437.845422:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518088960:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829437.845443:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829437.845448:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829437.845451:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829437.845459:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2444 00000100:00080000:0.0:1787829437.845464:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829437.845469:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829437.845477:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829437.845479:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829437.845498:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518089472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829437.845500:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff8b2e88040b40 x1874672518088960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829437.845555:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518088960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829437.845571:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.845576:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829437.845579:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518088960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829437.845584:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518089088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829437.845604:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829437.845613:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff8b2e880421c0 x1874672518089088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829437.845621:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518089088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829437.845627:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.845629:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829437.845631:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518089088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829437.845664:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518089216:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829437.845679:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829437.845687:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff8b2e88042d00 x1874672518089216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829437.845695:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518089216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829437.845702:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.845704:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829437.845706:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518089216:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829437.845710:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518089344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829437.845721:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829437.845727:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff8b2e88040780 x1874672518089344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829437.845739:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518089344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829437.845745:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.845747:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829437.845748:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518089344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829437.845752:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518089472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829437.845762:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829437.845777:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829437/real 1787829437] req@ffff8b2e88041e00 x1874672518089472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829437.845788:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518089472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829462 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829437.845794:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829437.845796:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829437.845798:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518089472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829438.164758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829438.676756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829438.676875:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829438.868734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829438.868735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829439.188778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829439.700729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829439.700834:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829439.892777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829439.892777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829440.212722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829440.724769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829440.724855:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829440.916737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829440.916802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829441.236776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829441.748766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829441.748799:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829441.940733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829441.940734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829442.260741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829442.772733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829442.772827:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829442.964773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829442.964773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829442.964813:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829442.964820:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829442.964825:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829442.964837:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2449 00000100:00080000:2.0:1787829442.964851:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829442.964861:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829442.964912:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518089600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829442.964940:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829442.964943:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829442.964944:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829442.964948:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2449 00000100:00080000:2.0:1787829442.964951:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829442.964959:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829442.964969:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829442.964982:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518089728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829442.965005:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518089600:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829442.965010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829442.965017:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829442.965023:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829442.965028:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2449 00000100:00080000:2.0:1787829442.965031:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829442.965034:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829442.965038:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829442.965051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518089856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829442.965059:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829442.965061:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829442.965063:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829442.965065:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2449 00000100:00080000:2.0:1787829442.965078:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829442.965081:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829442.965084:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829442.965091:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518089984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829442.965101:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829442.965108:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829442.965112:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829442.965114:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829442.965118:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2449 00000100:00080000:2.0:1787829442.965120:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829442.965125:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829442.965127:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829442.965136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518090112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829442.965143:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829442.965146:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829442.965148:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829442.965151:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2449 00000100:00080000:2.0:1787829442.965154:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829442.965158:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829442.965162:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829442.965168:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff8b2e8801c3c0 x1874672518089600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829442.965173:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518090240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829442.965217:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829442.965224:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829442.965227:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829442.965232:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2449 00000100:00080000:2.0:1787829442.965238:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829442.965246:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829442.965255:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829442.965265:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518090368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829442.965281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518089600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.965292:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.965299:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829442.965305:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518089600:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829442.965330:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518089728:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829442.965359:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829442.965374:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff8b2e8801de00 x1874672518089728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.965385:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518089728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.965391:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.965394:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829442.965396:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518089728:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829442.965401:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518089856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829442.965424:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829442.965442:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff8b2e8801cb40 x1874672518089856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.965452:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518089856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.965462:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.965464:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829442.965466:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518089856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829442.965470:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518089984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829442.965482:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829442.965489:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff8b2e8801f0c0 x1874672518089984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.965496:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518089984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.965501:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.965505:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829442.965508:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518089984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829442.965515:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518090112:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829442.965539:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829442.965551:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff8b2e8801c780 x1874672518090112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.965566:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518090112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.965572:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.965574:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829442.965577:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518090112:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829442.965581:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518090240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829442.965597:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829442.965605:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff8b2e8801ed00 x1874672518090240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.965617:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518090240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.965627:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.965631:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829442.965634:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518090240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829442.965641:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518090368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829442.965693:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829442.965702:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829442/real 1787829442] req@ffff8b2e8801d680 x1874672518090368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829442.965711:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518090368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829467 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829442.965717:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829442.965719:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829442.965721:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518090368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829443.284755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829443.796763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829443.796849:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829443.988743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829443.988754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829444.308772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829444.820749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829444.820803:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829445.012764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829445.012770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829445.332795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829445.844741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829445.844811:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829446.036764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829446.036764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829446.356737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829446.868726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829446.868790:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829447.060776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829447.060819:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829447.380785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829447.892766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829447.892843:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829448.084802:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829448.084805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829448.084805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829448.084816:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829448.084820:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829448.084829:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2454 00000100:00080000:2.0:1787829448.084839:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829448.084846:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829448.084883:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518090496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829448.084908:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829448.084911:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829448.084912:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829448.084915:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2454 00000100:00080000:2.0:1787829448.084917:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829448.084919:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829448.084927:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829448.084937:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518090624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829448.084945:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829448.084962:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518090496:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829448.084963:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829448.084967:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829448.084973:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2454 00000100:00080000:2.0:1787829448.084976:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829448.084979:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829448.084983:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829448.085000:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518090752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829448.085012:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829448.085016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829448.085019:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829448.085023:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2454 00000100:00080000:2.0:1787829448.085027:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829448.085034:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829448.085035:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787829448.085041:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829448.085056:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518090880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829448.085063:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829448.085065:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829448.085067:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829448.085070:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2454 00000100:00080000:2.0:1787829448.085073:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829448.085076:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829448.085080:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829448.085087:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829448/real 1787829448] req@ffff8b2e8801da40 x1874672518090496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829448.085090:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518091008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829448.085122:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829448.085125:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829448.085126:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829448.085130:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2454 00000100:00080000:2.0:1787829448.085132:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829448.085134:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518090496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829448.085136:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829448.085142:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829448.085153:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518091136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829448.085157:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829448.085160:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829448.085164:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829448.085166:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829448.085167:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829448.085171:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2454 00000100:00080000:2.0:1787829448.085174:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829448.085177:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518090496:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829448.085180:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829448.085186:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829448.085194:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518091264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829448.085210:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518090624:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829448.085238:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829448.085253:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829448/real 1787829448] req@ffff8b2e8801fc00 x1874672518090624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829448.085268:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518090624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829448.085277:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829448.085281:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829448.085284:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518090624:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829448.085292:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518090752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829448.085315:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829448.085328:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829448/real 1787829448] req@ffff8b2e8801f480 x1874672518090752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829448.085341:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518090752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829448.085351:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829448.085355:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829448.085358:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518090752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829448.085366:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518090880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829448.085387:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829448.085401:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829448/real 1787829448] req@ffff8b2e8801cf00 x1874672518090880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829448.085415:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518090880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829448.085424:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829448.085428:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829448.085432:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518090880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829448.085439:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518091008:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829448.085460:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829448.085473:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829448/real 1787829448] req@ffff8b2e8801c000 x1874672518091008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829448.085487:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518091008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829448.085497:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829448.085500:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829448.085504:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518091008:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829448.085510:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518091136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829448.085531:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829448.085543:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829448/real 1787829448] req@ffff8b2e8801e580 x1874672518091136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829448.085555:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518091136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829448.085564:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829448.085567:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829448.085571:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518091136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829448.085577:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518091264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829448.085597:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829448.085608:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829448/real 1787829448] req@ffff8b2e8801e940 x1874672518091264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829448.085621:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518091264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829473 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829448.085629:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829448.085632:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829448.085636:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518091264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829448.404724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829448.916762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829448.916813:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829449.108767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829449.108767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829449.428774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829449.940771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829449.940828:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829450.132789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829450.132789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829450.452723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829450.964786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829450.964867:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829451.156778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829451.156789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829451.476765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829451.988752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829451.988818:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829452.180761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829452.180767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829452.500779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829453.012728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829453.012843:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829453.204743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829453.204775:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829453.204780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829453.204784:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829453.204790:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829453.204800:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2459 00000100:00080000:2.0:1787829453.204810:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829453.204817:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829453.204856:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518091392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829453.204878:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829453.204881:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829453.204882:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829453.204885:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2459 00000100:00080000:2.0:1787829453.204888:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829453.204891:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829453.204899:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829453.204909:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518091520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829453.204931:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518091392:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829453.204937:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829453.204942:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829453.204943:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829453.204947:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2459 00000100:00080000:2.0:1787829453.204949:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829453.204953:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829453.204958:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829453.204970:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518091648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829453.204978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829453.204980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829453.204981:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829453.204984:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2459 00000100:00080000:2.0:1787829453.204988:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829453.204991:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829453.204995:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829453.205002:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518091776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829453.205008:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829453.205012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829453.205013:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829453.205016:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2459 00000100:00080000:2.0:1787829453.205018:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829453.205020:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829453.205023:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829453.205030:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829453.205031:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518091904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829453.205044:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829453.205046:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829453.205047:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829453.205052:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2459 00000100:00080000:2.0:1787829453.205058:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829453.205060:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829453.205064:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829453.205073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518092032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829453.205082:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829453.205084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829453.205086:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829453.205090:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2459 00000100:00080000:2.0:1787829453.205092:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829453.205099:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829453.205101:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829453/real 1787829453] req@ffff8b2e8801e940 x1874672518091392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829453.205104:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829453.205120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518092160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829453.205151:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518091392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829453.205167:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829453.205172:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829453.205177:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518091392:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829453.205203:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518091520:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829453.205229:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829453.205243:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829453/real 1787829453] req@ffff8b2e8801e580 x1874672518091520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829453.205257:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518091520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829453.205277:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829453.205281:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829453.205289:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518091520:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829453.205299:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518091648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829453.205320:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829453.205332:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829453/real 1787829453] req@ffff8b2e8801c000 x1874672518091648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829453.205344:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518091648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829453.205353:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829453.205366:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829453.205370:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518091648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829453.205377:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518091776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829453.205396:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829453.205407:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829453/real 1787829453] req@ffff8b2e8801cf00 x1874672518091776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829453.205419:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518091776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829453.205427:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829453.205431:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829453.205435:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518091776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829453.205442:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518091904:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829453.205459:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829453.205470:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829453/real 1787829453] req@ffff8b2e8801f480 x1874672518091904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829453.205483:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518091904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829453.205491:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829453.205495:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829453.205498:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518091904:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829453.205504:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518092032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829453.205522:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829453.205532:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829453/real 1787829453] req@ffff8b2e8801fc00 x1874672518092032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829453.205544:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518092032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829453.205552:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829453.205556:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829453.205560:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518092032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829453.205566:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518092160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829453.205583:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829453.205593:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829453/real 1787829453] req@ffff8b2e8801da40 x1874672518092160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829453.205605:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518092160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829478 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829453.205620:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829453.205623:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829453.205626:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518092160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829453.524724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829454.036756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829454.036840:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829454.228742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829454.228744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829454.548777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829455.060766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829455.060855:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829455.252706:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829455.252716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829455.572733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829456.084762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829456.084839:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829456.276752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829456.276771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829456.596770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829457.108731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829457.108816:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829457.300717:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829457.300730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829457.620735:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829458.132759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829458.132821:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829458.324736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829458.324774:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829458.324787:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829458.324792:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829458.324794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829458.324805:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2464 00000100:00080000:0.0:1787829458.324818:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829458.324826:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829458.324865:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518092288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829458.324912:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829458.324916:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829458.324918:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829458.324923:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2464 00000100:00080000:0.0:1787829458.324927:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829458.324931:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829458.324942:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829458.324960:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518092416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829458.324976:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829458.324976:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518092288:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829458.324989:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829458.324991:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829458.324996:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2464 00000100:00080000:0.0:1787829458.325001:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829458.325007:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829458.325013:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829458.325032:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518092544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829458.325043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829458.325047:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829458.325050:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829458.325055:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2464 00000100:00080000:0.0:1787829458.325060:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829458.325065:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829458.325071:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829458.325084:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518092672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829458.325096:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829458.325101:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829458.325103:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829458.325107:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2464 00000100:00080000:0.0:1787829458.325111:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829458.325116:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829458.325122:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518092416:192.168.203.101@tcp:38:kworker.0 00000008:00100000:0.0:1787829458.325123:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829458.325146:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518092800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829458.325160:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829458.325164:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518092544:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829458.325169:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829458.325173:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829458.325181:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2464 00000100:00100000:3.0:1787829458.325188:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518092672:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829458.325189:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829458.325204:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829458.325212:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829458.325218:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518092800:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829458.325247:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518092928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829458.325267:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829458.325271:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829458.325274:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829458.325280:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2464 00000100:00080000:0.0:1787829458.325286:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829458.325295:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518092928:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829458.325296:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829458.325308:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829458.325331:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518093056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829458.325372:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518093056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829458.644766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829459.156727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829459.156813:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829459.348772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829459.348861:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829459.668791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829460.180784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829460.180834:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829460.372755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829460.372755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829460.692780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829461.204726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829461.204810:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829461.396777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829461.396798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:0.0:1787829461.396979:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:0.0:1787829461.397023:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829461.397040:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829461.397079:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829461.397088:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829461.397100:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829461.397112:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829461.397123:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829461.397130:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829461] req@ffff8b2e88041e00 x1874672518092416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:0.0:1787829461.397138:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829461.397171:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518092416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829461.397187:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829461.397192:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829461.397197:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518092416:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829461.397205:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829461] req@ffff8b2e88040780 x1874672518092544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829461.397220:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518092544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829461.397229:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829461.397233:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829461.397238:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518092544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829461.397245:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829461] req@ffff8b2e88042d00 x1874672518092672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829461.397258:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518092672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829461.397266:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829461.397270:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829461.397273:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518092672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829461.397280:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829461] req@ffff8b2e88042580 x1874672518092288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829461.397292:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518092288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829461.397300:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829461.397303:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829461.397307:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518092288:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829461.397334:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829461] req@ffff8b2e880412c0 x1874672518093056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829461.397346:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518093056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829461.397354:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829461.397357:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829461.397361:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518093056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829461.397367:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829461] req@ffff8b2e88040b40 x1874672518092928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829461.397377:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518092928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829461.397383:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829461.397386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829461.397389:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518092928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829461.397394:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829458/real 1787829461] req@ffff8b2e880421c0 x1874672518092800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829461.397404:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518092800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829483 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829461.397412:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829461.397416:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829461.397419:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518092800:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829461.716755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829462.228758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829462.228834:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829462.420698:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829462.420714:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829462.740771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829463.252785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829463.252882:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829463.444765:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829463.444773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829463.444779:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829463.444785:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829463.444798:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2469 00000100:00080000:0.0:1787829463.444813:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829463.444820:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829463.444856:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829463.444878:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518093184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829463.444905:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829463.444910:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829463.444912:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829463.444917:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2469 00000100:00080000:0.0:1787829463.444922:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829463.444927:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829463.444940:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829463.444956:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518093184:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829463.444959:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518093312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829463.444976:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829463.444980:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829463.444982:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829463.444987:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2469 00000100:00080000:0.0:1787829463.444992:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829463.444998:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829463.445005:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829463.445021:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518093440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829463.445032:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829463.445036:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829463.445037:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829463.445040:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2469 00000100:00080000:0.0:1787829463.445044:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829463.445047:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829463.445051:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829463.445058:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829463.445073:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518093568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829463.445084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829463.445088:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829463.445090:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829463.445095:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2469 00000100:00080000:0.0:1787829463.445100:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829463.445106:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff8b2e88043840 x1874672518093184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829463.445107:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829463.445116:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829463.445136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518093696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829463.445138:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518093184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829463.445162:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829463.445169:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829463.445169:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829463.445174:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829463.445179:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829463.445185:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2469 00000100:00100000:3.0:1787829463.445190:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518093184:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829463.445192:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829463.445202:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829463.445209:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829463.445223:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518093824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829463.445230:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518093312:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829463.445242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829463.445247:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829463.445250:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829463.445255:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2469 00000100:00080000:0.0:1787829463.445261:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829463.445269:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829463.445270:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829463.445280:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829463.445292:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff8b2e88040f00 x1874672518093312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829463.445299:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518093952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829463.445321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518093312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829463.445333:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829463.445338:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829463.445343:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518093312:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829463.445351:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518093440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829463.445377:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829463.445389:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff8b2e88041a40 x1874672518093440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829463.445402:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518093440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829463.445411:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829463.445414:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829463.445417:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518093440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829463.445424:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518093568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829463.445443:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829463.445454:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff8b2e88040000 x1874672518093568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829463.445466:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518093568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829463.445475:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829463.445479:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829463.445482:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518093568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829463.445488:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518093696:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829463.445509:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829463.445520:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff8b2e880430c0 x1874672518093696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829463.445533:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518093696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829463.445541:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829463.445545:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829463.445548:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518093696:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829463.445703:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518093824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829463.445729:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829463.445742:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff8b2e88043480 x1874672518093824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829463.445754:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518093824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829463.445762:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829463.445765:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829463.445769:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518093824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829463.445775:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518093952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829463.445791:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829463.445801:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829463/real 1787829463] req@ffff8b2e88042940 x1874672518093952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829463.445812:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518093952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829488 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829463.445826:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829463.445829:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829463.445832:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518093952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829463.764766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829464.276744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829464.276796:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829464.468793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829464.468910:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829464.788781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829465.300758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829465.300849:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829465.492760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829465.492760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829465.812758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787829466.324802:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829466.324893:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787829466.516745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829466.516818:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829466.836730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829467.348766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829467.348861:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829467.540758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829467.540765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829467.860723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829468.372733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829468.372824:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829468.564746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829468.564747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829468.564757:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829468.564766:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829468.564772:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829468.564783:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2474 00000100:00080000:3.0:1787829468.564798:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829468.564817:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829468.564860:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518094080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829468.564885:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829468.564888:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829468.564890:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829468.564893:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2474 00000100:00080000:3.0:1787829468.564897:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829468.564901:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829468.564911:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829468.564922:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518094080:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829468.564923:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518094208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829468.564937:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829468.564940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829468.564942:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829468.564945:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2474 00000100:00080000:3.0:1787829468.564947:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829468.564950:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829468.564954:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829468.564961:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518094336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829468.564967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829468.564969:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829468.564971:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829468.564974:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2474 00000100:00080000:3.0:1787829468.564978:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829468.564980:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829468.564983:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829468.564998:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518094464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829468.565004:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829468.565007:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829468.565009:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829468.565014:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829468.565016:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2474 00000100:00080000:3.0:1787829468.565021:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829468.565024:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829468.565027:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829468.565036:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518094592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829468.565043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829468.565045:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829468.565047:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829468.565049:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2474 00000100:00080000:3.0:1787829468.565051:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829468.565054:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829468.565057:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829468.565065:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518094720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829468.565071:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829468.565074:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829468.565077:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829468.565079:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff8b2e88039a40 x1874672518094080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829468.565081:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2474 00000100:00080000:3.0:1787829468.565085:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829468.565090:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829468.565097:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829468.565105:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518094848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829468.565125:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518094080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829468.565137:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829468.565141:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829468.565145:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518094080:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829468.565166:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518094208:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829468.565186:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829468.565197:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff8b2e88039e00 x1874672518094208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829468.565206:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518094208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829468.565212:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829468.565214:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829468.565216:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518094208:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829468.565220:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518094336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829468.565232:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829468.565238:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff8b2e8803b480 x1874672518094336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829468.565246:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518094336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829468.565251:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829468.565253:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829468.565255:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518094336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829468.565259:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518094464:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829468.565270:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829468.565276:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff8b2e88039680 x1874672518094464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829468.565283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518094464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829468.565289:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829468.565290:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829468.565292:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518094464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829468.565295:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518094592:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829468.565305:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829468.565311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff8b2e880392c0 x1874672518094592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829468.565318:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518094592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829468.565324:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829468.565326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829468.565328:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518094592:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829468.565331:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518094720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829468.565342:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829468.565348:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff8b2e880383c0 x1874672518094720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829468.565367:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518094720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829468.565374:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829468.565376:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829468.565378:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518094720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829468.565382:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518094848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829468.565394:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829468.565401:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829468/real 1787829468] req@ffff8b2e8803a580 x1874672518094848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829468.565408:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518094848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829493 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829468.565414:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829468.565416:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829468.565418:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518094848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829468.884751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829469.396753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829469.396780:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829469.588772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829469.588772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829469.908770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829470.420818:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829470.420841:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829470.612726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829470.612758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829470.932767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829471.444739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829471.444803:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829471.636729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829471.636796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829471.956751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829472.468744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829472.468752:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829472.660765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829472.660772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829472.980773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829473.492767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829473.492945:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829473.684751:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829473.684761:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787829473.684768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829473.684768:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829473.684791:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2480 00000100:00100000:0.0:1787829473.684802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829473.684806:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829473.684818:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829473.684860:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518094976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829473.684884:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829473.684887:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829473.684889:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829473.684892:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2480 00000100:00080000:3.0:1787829473.684895:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829473.684897:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829473.684907:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829473.684916:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518095104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829473.684925:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829473.684928:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829473.684930:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829473.684934:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518094976:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829473.684935:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2480 00000100:00080000:3.0:1787829473.684940:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829473.684946:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829473.684949:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829473.684958:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518095232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829473.684966:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829473.684968:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829473.684970:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829473.684972:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2480 00000100:00080000:3.0:1787829473.684974:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829473.684976:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829473.684979:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829473.684987:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518095360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829473.684993:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829473.684995:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829473.684996:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829473.684999:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2480 00000100:00080000:3.0:1787829473.685002:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829473.685005:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829473.685007:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829473.685014:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518095488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829473.685020:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829473.685023:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829473.685024:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829473.685029:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829473.685032:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2480 00000100:00080000:3.0:1787829473.685036:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829473.685039:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829473.685042:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829473.685050:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518095616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829473.685060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829473.685063:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829473.685065:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829473.685069:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2480 00000100:00080000:3.0:1787829473.685074:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829473.685079:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829473.685085:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829473.685089:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff8b2e88038f00 x1874672518094976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829473.685102:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518095744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829473.685149:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518094976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829473.685159:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829473.685171:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829473.685175:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518094976:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829473.685198:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518095104:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829473.685219:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829473.685229:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff8b2e8803a1c0 x1874672518095104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829473.685238:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518095104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829473.685244:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829473.685246:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829473.685249:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518095104:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829473.685252:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518095232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829473.685264:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829473.685271:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff8b2e8803b0c0 x1874672518095232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829473.685278:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518095232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829473.685288:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829473.685290:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829473.685292:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518095232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829473.685295:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518095360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829473.685305:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829473.685311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff8b2e8803b840 x1874672518095360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829473.685324:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518095360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829473.685330:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829473.685332:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829473.685334:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518095360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829473.685338:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518095488:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829473.685350:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829473.685357:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff8b2e8803a580 x1874672518095488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829473.685365:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518095488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829473.685370:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829473.685372:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829473.685374:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518095488:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829473.685378:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518095616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829473.685389:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829473.685394:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff8b2e880383c0 x1874672518095616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829473.685402:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518095616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829473.685407:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829473.685409:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829473.685411:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518095616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829473.685414:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518095744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829473.685425:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829473.685431:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829473/real 1787829473] req@ffff8b2e880392c0 x1874672518095744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829473.685438:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518095744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829498 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829473.685443:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829473.685450:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829473.685453:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518095744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829474.004791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829474.516809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829474.516819:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829474.708814:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829474.708828:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829475.028779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829475.540786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829475.540828:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829475.732813:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829475.732824:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829476.052784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829476.564747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829476.564766:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829476.756777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829476.756777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829477.076727:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829477.588772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829477.588818:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829477.780752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829477.780752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829478.100782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829478.612760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829478.612782:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829478.804795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829478.804797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829478.804836:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.804842:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829478.804845:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.804853:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2485 00000100:00080000:2.0:1787829478.804862:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.804873:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829478.804904:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518095872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829478.804924:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.804928:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829478.804930:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.804933:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2485 00000100:00080000:2.0:1787829478.804937:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.804941:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829478.804951:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829478.804959:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518095872:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829478.804966:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518096000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829478.804975:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.804977:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829478.804978:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.804981:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2485 00000100:00080000:2.0:1787829478.804984:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.804986:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829478.804989:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829478.804998:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518096128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829478.805004:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.805006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829478.805008:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.805009:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2485 00000100:00080000:2.0:1787829478.805011:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.805013:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829478.805016:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829478.805019:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829478.805025:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518096256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829478.805035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.805038:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829478.805040:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.805044:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2485 00000100:00080000:2.0:1787829478.805047:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.805050:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829478.805055:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829478.805068:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff8b2e8801d680 x1874672518095872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829478.805069:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518096384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829478.805094:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.805096:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829478.805098:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829478.805100:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518095872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829478.805104:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2485 00000100:00080000:2.0:1787829478.805109:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.805114:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829478.805115:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829478.805121:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:2.0:1787829478.805121:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829478.805132:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518096512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829478.805134:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518095872:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829478.805144:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829478.805147:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829478.805149:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829478.805151:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2485 00000100:00080000:2.0:1787829478.805153:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829478.805155:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829478.805158:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518096000:192.168.203.101@tcp:38:kworker.0 00000008:00100000:2.0:1787829478.805158:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829478.805166:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518096640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829478.805179:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829478.805192:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff8b2e8801ed00 x1874672518096000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829478.805204:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518096000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829478.805212:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829478.805215:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829478.805218:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518096000:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829478.805226:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518096128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829478.805243:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829478.805253:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff8b2e8801c780 x1874672518096128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829478.805264:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518096128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829478.805272:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829478.805275:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829478.805278:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518096128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829478.805284:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518096256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829478.805303:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829478.805313:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff8b2e8801f0c0 x1874672518096256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829478.805324:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518096256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829478.805332:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829478.805335:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829478.805338:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518096256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829478.805343:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518096384:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829478.805357:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829478.805364:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff8b2e8801cb40 x1874672518096384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829478.805376:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518096384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829478.805383:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829478.805386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829478.805390:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518096384:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829478.805395:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518096512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829478.805413:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829478.805420:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff8b2e8801de00 x1874672518096512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829478.805427:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518096512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829478.805432:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829478.805433:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829478.805435:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518096512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829478.805447:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518096640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829478.805465:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829478.805471:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829478/real 1787829478] req@ffff8b2e8801c3c0 x1874672518096640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829478.805478:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518096640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829503 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829478.805486:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829478.805488:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829478.805490:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518096640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829479.124778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787829479.636809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829479.636811:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829479.828709:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829479.828790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829480.148775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829480.660763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829480.661009:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829480.852761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829480.852762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829481.172802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829481.684752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829481.684803:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829481.876758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829481.876768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829482.196779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829482.708768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829482.708859:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829482.900768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829482.900768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829483.220759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829483.732741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829483.732819:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829483.924791:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829483.924813:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829483.924822:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829483.924832:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829483.924837:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829483.924848:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2490 00000100:00080000:0.0:1787829483.924861:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829483.924868:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829483.924909:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518096768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829483.924942:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829483.924947:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829483.924950:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829483.924957:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2490 00000100:00080000:0.0:1787829483.924963:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829483.924968:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829483.924973:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518096768:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829483.924983:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829483.925006:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518096896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829483.925018:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829483.925021:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829483.925023:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829483.925027:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2490 00000100:00080000:0.0:1787829483.925031:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829483.925035:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829483.925036:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829483.925046:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829483.925059:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518097024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829483.925067:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829483.925070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829483.925072:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829483.925075:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2490 00000100:00080000:0.0:1787829483.925078:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829483.925083:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829483.925085:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff8b2e880403c0 x1874672518096768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829483.925088:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829483.925107:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518097152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829483.925115:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518096768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829483.925123:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829483.925139:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829483.925147:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829483.925148:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829483.925171:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829483.925175:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2490 00000100:00100000:3.0:1787829483.925180:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518096768:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829483.925182:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829483.925188:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829483.925194:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829483.925202:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518096896:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829483.925207:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518097280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829483.925218:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829483.925221:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829483.925222:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829483.925226:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2490 00000100:00080000:0.0:1787829483.925229:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829483.925235:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829483.925238:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829483.925242:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829483.925255:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518097408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829483.925255:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff8b2e88042940 x1874672518096896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829483.925313:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829483.925318:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829483.925319:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518096896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829483.925321:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829483.925333:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2490 00000100:00080000:0.0:1787829483.925341:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829483.925344:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829483.925348:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829483.925354:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000008:00100000:0.0:1787829483.925359:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829483.925361:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518096896:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829483.925373:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518097024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829483.925376:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518097536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829483.925398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829483.925417:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff8b2e88043480 x1874672518097024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829483.925430:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518097024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829483.925444:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829483.925448:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829483.925450:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518097024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829483.925455:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518097152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829483.925470:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829483.925477:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff8b2e880430c0 x1874672518097152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829483.925483:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518097152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829483.925488:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829483.925490:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829483.925492:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518097152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829483.925503:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518097280:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829483.925512:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829483.925517:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff8b2e88040000 x1874672518097280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829483.925524:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518097280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829483.925529:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829483.925530:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829483.925533:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518097280:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829483.925537:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518097408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829483.925545:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829483.925550:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff8b2e88041a40 x1874672518097408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829483.925556:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518097408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829483.925560:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829483.925562:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829483.925563:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518097408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829483.925567:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518097536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829483.925575:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829483.925580:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829483/real 1787829483] req@ffff8b2e88040f00 x1874672518097536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829483.925586:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518097536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829508 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829483.925590:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829483.925591:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829483.925593:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518097536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829484.244765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829484.756798:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829484.756839:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829484.948797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829484.948801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829485.268741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829485.780748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829485.780812:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829485.972777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829485.972784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829486.292786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829486.804824:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829486.804858:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829486.996790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829486.996793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829487.316759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829487.828744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829487.828821:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829488.020735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829488.020735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829488.340783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829488.852724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829488.852799:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829489.044782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829489.044782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829489.044810:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829489.044819:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829489.044825:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829489.044839:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2495 00000100:00080000:1.0:1787829489.044854:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829489.044864:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829489.044913:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518097664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829489.044945:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829489.044949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829489.044951:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829489.044964:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2495 00000100:00080000:1.0:1787829489.044969:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829489.044974:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829489.044985:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518097664:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787829489.044986:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829489.045008:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518097792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829489.045021:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829489.045026:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829489.045028:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829489.045033:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2495 00000100:00080000:1.0:1787829489.045038:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829489.045042:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829489.045049:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829489.045061:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829489.045064:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518097920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829489.045077:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829489.045081:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829489.045083:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829489.045087:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2495 00000100:00080000:1.0:1787829489.045091:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829489.045095:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829489.045100:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829489.045111:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518098048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829489.045116:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829489/real 1787829489] req@ffff8b2e88074f00 x1874672518097664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829489.045135:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829489.045140:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829489.045142:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829489.045146:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518097664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829489.045151:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2495 00000100:00080000:1.0:1787829489.045158:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829489.045164:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829489.045164:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829489.045171:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829489.045171:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829489.045178:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518097664:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829489.045194:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518098176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829489.045203:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518097792:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829489.045210:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829489.045215:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829489.045218:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829489.045224:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2495 00000100:00080000:1.0:1787829489.045229:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829489.045229:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829489.045236:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829489.045243:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829489.045243:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829489/real 1787829489] req@ffff8b2e88076940 x1874672518097792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829489.045262:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518098304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829489.045262:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518097792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829489.045294:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829489.045297:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829489.045300:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829489.045304:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829489.045308:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518097792:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829489.045309:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829489.045318:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2495 00000100:00100000:3.0:1787829489.045319:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518097920:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829489.045325:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829489.045331:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829489.045341:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829489.045347:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829489.045358:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518098432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829489.045359:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829489/real 1787829489] req@ffff8b2e88077480 x1874672518097920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829489.045392:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518097920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829489.045399:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829489.045401:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829489.045404:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518097920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829489.045410:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518098048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829489.045423:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829489.045430:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829489/real 1787829489] req@ffff8b2e88075a40 x1874672518098048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829489.045438:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518098048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829489.045443:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829489.045446:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829489.045448:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518098048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829489.045458:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518098176:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829489.045468:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829489.045476:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829489/real 1787829489] req@ffff8b2e88075680 x1874672518098176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829489.045483:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518098176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829489.045489:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829489.045491:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829489.045493:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518098176:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829489.045496:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518098304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829489.045506:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829489.045512:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829489/real 1787829489] req@ffff8b2e88077c00 x1874672518098304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829489.045520:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518098304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829489.045525:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829489.045527:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829489.045529:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518098304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829489.045532:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518098432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829489.045541:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829489.045547:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829489/real 1787829489] req@ffff8b2e88074780 x1874672518098432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829489.045554:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518098432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829514 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829489.045560:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829489.045562:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829489.045564:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518098432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829489.364748:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829489.876774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829489.876810:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829490.068774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829490.068795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829490.388760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829490.900739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829490.900767:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829491.092763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829491.092841:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829491.412772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829491.924764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829491.924859:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829492.116772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829492.116785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829492.436771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829492.948725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829492.948814:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829493.140755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829493.140755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829493.460770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829493.972719:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829493.972759:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829494.164721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829494.164728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829494.164728:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829494.164734:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829494.164739:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829494.164746:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2500 00000100:00080000:1.0:1787829494.164756:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829494.164760:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829494.164790:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518098560/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829494.164807:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829494.164809:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829494.164810:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829494.164813:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2500 00000100:00080000:1.0:1787829494.164815:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829494.164818:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829494.164825:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829494.164832:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518098560:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829494.164834:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518098688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829494.164843:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829494.164846:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829494.164847:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829494.164849:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2500 00000100:00080000:1.0:1787829494.164851:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829494.164853:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829494.164856:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829494.164863:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518098816/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829494.164873:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829494.164875:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829494.164877:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829494.164878:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2500 00000100:00080000:1.0:1787829494.164880:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829494.164882:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829494.164885:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829494.164890:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829494.164893:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518098944/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829494.164899:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829494.164902:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829494.164903:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829494.164905:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2500 00000100:00080000:1.0:1787829494.164907:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829494.164910:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829494.164912:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829494.164918:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518099072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829494.164923:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829494.164925:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829494.164926:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829494.164929:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2500 00000100:00080000:1.0:1787829494.164932:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829494.164934:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829494.164937:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829494.164942:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829494/real 1787829494] req@ffff8b2e88077840 x1874672518098560/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829494.164947:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518099200/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829494.164966:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829494.164969:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829494.164970:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829494.164976:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518098560/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829494.164978:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2500 00000100:00080000:1.0:1787829494.164984:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829494.164990:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829494.164991:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829494.164993:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829494.164997:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829494.165001:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518098560:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829494.165004:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518099328/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829494.165022:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518098688:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829494.165040:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829494.165049:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829494/real 1787829494] req@ffff8b2e88076580 x1874672518098688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829494.165061:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518098688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829494.165066:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829494.165068:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829494.165070:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518098688:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829494.165073:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518098816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829494.165083:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829494.165089:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829494/real 1787829494] req@ffff8b2e88076d00 x1874672518098816/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829494.165096:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518098816/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829494.165100:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829494.165102:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829494.165103:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518098816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829494.165106:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518098944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829494.165115:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829494.165122:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829494/real 1787829494] req@ffff8b2e880770c0 x1874672518098944/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829494.165132:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518098944/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829494.165145:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829494.165148:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829494.165151:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518098944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829494.165156:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518099072:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829494.165175:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829494.165186:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829494/real 1787829494] req@ffff8b2e88074000 x1874672518099072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829494.165198:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518099072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829494.165206:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829494.165208:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829494.165211:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518099072:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829494.165217:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518099200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829494.165236:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829494.165247:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829494/real 1787829494] req@ffff8b2e880761c0 x1874672518099200/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829494.165259:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518099200/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829494.165267:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829494.165269:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829494.165273:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518099200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829494.165279:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518099328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829494.165297:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829494.165308:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829494/real 1787829494] req@ffff8b2e88075e00 x1874672518099328/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829494.165320:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518099328/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829519 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829494.165327:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829494.165330:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829494.165333:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518099328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829494.484725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829494.996763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829494.996812:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829495.188733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829495.188733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829495.508743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829496.020729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829496.020774:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829496.212739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829496.212739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829496.532705:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829497.044750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829497.044775:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829497.236718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829497.236723:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829497.556755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829498.068701:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829498.068785:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829498.260728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829498.260728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829498.580716:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829499.092747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829499.092769:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829499.284713:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829499.284724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829499.284740:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829499.284747:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829499.284752:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829499.284761:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2505 00000100:00080000:1.0:1787829499.284772:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829499.284780:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829499.284818:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518099456/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829499.284843:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829499.284847:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829499.284848:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829499.284853:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2505 00000100:00080000:1.0:1787829499.284856:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829499.284861:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829499.284871:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829499.284872:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518099456:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829499.284890:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518099584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829499.284901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829499.284904:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829499.284906:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829499.284910:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2505 00000100:00080000:1.0:1787829499.284913:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829499.284916:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829499.284920:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829499.284930:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518099712/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829499.284931:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829499.284939:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829499.284943:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829499.284944:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829499.284946:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2505 00000100:00080000:1.0:1787829499.284948:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829499.284950:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829499.284952:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829499.284957:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518099840/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829499.284963:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829499.284965:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829499.284966:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829499.284969:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2505 00000100:00080000:1.0:1787829499.284970:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829499.284972:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829499.284975:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829499.284984:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518099968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829499.284988:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829499] req@ffff8b2e88074b40 x1874672518099456/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829499.285009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829499.285010:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518099456/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829499.285016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829499.285021:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829499.285029:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829499.285030:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2505 00000100:00080000:1.0:1787829499.285037:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829499.285042:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829499.285047:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787829499.285048:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829499.285056:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518099456:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829499.285063:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518100096/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829499.285072:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829499.285076:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829499.285078:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829499.285082:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518099584:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829499.285084:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2505 00000100:00080000:1.0:1787829499.285090:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829499.285094:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829499.285099:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829499.285111:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829499.285119:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518100224/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829499.285127:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829499] req@ffff8b2e88075e00 x1874672518099584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829499.285142:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518099584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829499.285151:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829499.285155:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829499.285158:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518099584:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829499.285165:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518099712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829499.285181:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829499.285192:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829499] req@ffff8b2e880761c0 x1874672518099712/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829499.285202:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518099712/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829499.285209:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829499.285212:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829499.285215:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518099712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829499.285220:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518099840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829499.285235:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829499.285246:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829499] req@ffff8b2e88074000 x1874672518099840/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829499.285257:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518099840/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829499.285265:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829499.285268:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829499.285270:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518099840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829499.285275:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518099968:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829499.285292:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829499.285302:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829499] req@ffff8b2e880770c0 x1874672518099968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829499.285312:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518099968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829499.285318:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829499.285321:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829499.285324:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518099968:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829499.285330:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518100096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829499.285346:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829499.285356:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829499] req@ffff8b2e88076d00 x1874672518100096/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829499.285367:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518100096/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829499.285374:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829499.285377:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829499.285381:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518100096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829499.285398:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518100224:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829499.285415:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829499.285425:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829499/real 1787829499] req@ffff8b2e88076580 x1874672518100224/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829499.285435:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518100224/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829524 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829499.285442:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829499.285445:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829499.285448:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518100224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829499.604765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829500.116716:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829500.116742:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829500.308743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829500.308743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829500.628771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829501.140703:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829501.140776:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829501.332760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829501.332760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829501.652765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829502.164861:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829502.164940:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829502.356789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829502.356792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829502.676757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829503.188750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829503.188775:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829503.380715:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829503.380716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829503.700766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829504.212777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829504.212843:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829504.404734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829504.404734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829504.404739:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.404747:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829504.404752:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.404762:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2510 00000100:00080000:1.0:1787829504.404776:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.404782:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829504.404816:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518100352/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829504.404839:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.404842:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829504.404843:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.404845:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2510 00000100:00080000:1.0:1787829504.404847:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.404849:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829504.404857:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829504.404865:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518100480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829504.404872:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.404874:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829504.404875:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.404878:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2510 00000100:00080000:1.0:1787829504.404879:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.404881:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829504.404884:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829504.404893:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518100608/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829504.404903:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518100352:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829504.404904:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.404916:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829504.404925:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.404929:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2510 00000100:00080000:1.0:1787829504.404931:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.404933:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829504.404936:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829504.404943:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518100736/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829504.404948:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.404950:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829504.404951:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.404954:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2510 00000100:00080000:1.0:1787829504.404956:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.404958:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829504.404960:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829504.404971:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518100864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829504.404976:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.404978:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829504.404979:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.404982:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2510 00000100:00080000:1.0:1787829504.404985:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.404988:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829504.404992:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829504.404996:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518100992/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829504.405002:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829504.405005:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829504.405006:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829504.405008:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2510 00000100:00080000:1.0:1787829504.405014:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829504.405016:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829504.405019:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829504.405028:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829504.405036:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518101120/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829504.405093:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff8b2e88077840 x1874672518100352/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829504.405117:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518100352/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829504.405131:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829504.405137:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829504.405142:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518100352:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829504.405151:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518100480:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829504.405182:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829504.405199:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff8b2e88074780 x1874672518100480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829504.405214:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518100480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829504.405225:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829504.405229:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829504.405233:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518100480:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829504.405264:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518100608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829504.405289:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829504.405303:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff8b2e88077c00 x1874672518100608/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829504.405317:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518100608/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829504.405327:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829504.405331:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829504.405334:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518100608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829504.405342:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518100736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829504.405364:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829504.405378:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff8b2e88075680 x1874672518100736/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829504.405391:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518100736/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829504.405401:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829504.405405:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829504.405409:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518100736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829504.405417:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518100864:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829504.405439:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829504.405452:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff8b2e88075a40 x1874672518100864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829504.405466:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518100864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829504.405477:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829504.405481:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829504.405484:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518100864:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829504.405492:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518100992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829504.405512:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829504.405525:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff8b2e88077480 x1874672518100992/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829504.405539:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518100992/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829504.405549:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829504.405552:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829504.405571:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518100992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829504.405579:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518101120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829504.405601:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829504.405614:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829504/real 1787829504] req@ffff8b2e88076940 x1874672518101120/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829504.405627:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518101120/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829529 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829504.405637:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829504.405641:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829504.405644:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518101120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829504.724756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829505.236756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829505.236785:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829505.428775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829505.428775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829505.748767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829506.260732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829506.260836:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829506.452799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829506.452799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829506.772715:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829507.284758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829507.284828:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829507.476764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829507.476806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829507.796764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829508.308755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829508.308887:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829508.500753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829508.500763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829508.820722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829509.332765:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829509.332850:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:1.0:1787829509.524745:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829509.524753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829509.524755:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787829509.524757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829509.524759:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.524772:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2515 00000100:00080000:1.0:1787829509.524782:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.524789:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829509.524827:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518101248/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829509.524854:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.524858:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829509.524860:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.524864:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2515 00000100:00080000:1.0:1787829509.524869:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.524873:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829509.524883:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829509.524893:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518101248:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829509.524901:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518101376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829509.524910:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.524912:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829509.524914:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.524917:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2515 00000100:00080000:1.0:1787829509.524921:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.524925:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829509.524930:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829509.524942:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518101504/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829509.524951:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.524954:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829509.524956:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.524960:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2515 00000100:00080000:1.0:1787829509.524963:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.524968:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829509.524975:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829509.524987:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829509.524989:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518101632/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829509.525001:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.525006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829509.525008:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.525016:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2515 00000100:00080000:1.0:1787829509.525020:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.525026:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829509.525033:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829509.525042:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff8b2e88074f00 x1874672518101248/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829509.525050:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518101760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829509.525065:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518101248/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829509.525066:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.525086:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829509.525089:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829509.525091:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829509.525095:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829509.525104:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518101248:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829509.525109:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2515 00000100:00080000:1.0:1787829509.525118:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829509.525123:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829509.525128:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829509.525137:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518101376:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829509.525145:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518101888/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829509.525161:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829509.525164:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787829509.525166:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829509.525171:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829509.525179:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2515 00000100:00100000:3.0:1787829509.525182:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff8b2e88076940 x1874672518101376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829509.525189:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829509.525196:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518101376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829509.525204:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829509.525211:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829509.525214:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787829509.525223:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829509.525224:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518101376:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829509.525234:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518101504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829509.525243:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518102016/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829509.525255:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829509.525264:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff8b2e88077480 x1874672518101504/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829509.525276:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518101504/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829509.525286:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829509.525289:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829509.525291:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518101504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829509.525295:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518101632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829509.525305:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829509.525311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff8b2e88075a40 x1874672518101632/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829509.525318:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518101632/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829509.525324:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829509.525325:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829509.525327:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518101632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829509.525331:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518101760:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829509.525341:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829509.525347:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff8b2e88075680 x1874672518101760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829509.525355:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518101760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829509.525360:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829509.525362:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829509.525364:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518101760:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829509.525376:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518101888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829509.525387:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829509.525393:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff8b2e88077c00 x1874672518101888/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829509.525401:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518101888/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829509.525406:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829509.525408:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829509.525410:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518101888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829509.525414:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518102016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829509.525423:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829509.525429:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829509/real 1787829509] req@ffff8b2e88074780 x1874672518102016/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829509.525437:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518102016/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829534 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829509.525442:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829509.525445:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829509.525446:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518102016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829509.844772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829510.356732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829510.356828:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829510.548737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829510.548771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829510.868746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829511.380753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829511.380831:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829511.572731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829511.572736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829511.892727:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829512.404789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829512.404836:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829512.596726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829512.596728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829512.916761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829513.428759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829513.428849:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829513.620761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829513.620772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829513.940759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829514.452758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829514.452811:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829514.644745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829514.644758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829514.644779:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829514.644791:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829514.644796:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829514.644808:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2521 00000100:00080000:2.0:1787829514.644822:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829514.644831:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829514.644878:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518102144/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829514.644905:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829514.644908:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829514.644910:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829514.644914:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2521 00000100:00080000:2.0:1787829514.644919:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829514.644923:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829514.644937:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518102144:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787829514.644938:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829514.644959:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518102272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829514.644970:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829514.644974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829514.644976:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829514.644980:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2521 00000100:00080000:2.0:1787829514.645000:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829514.645003:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829514.645009:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829514.645016:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829514.645025:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518102400/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829514.645036:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829514.645038:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829514.645041:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829514.645046:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2521 00000100:00080000:2.0:1787829514.645052:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829514.645055:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829514.645060:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829514.645064:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff8b2e8801c3c0 x1874672518102144/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829514.645073:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518102528/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829514.645082:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829514.645083:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518102144/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829514.645090:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829514.645098:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829514.645115:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2521 00000100:00080000:3.0:1787829514.645119:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829514.645120:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829514.645126:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829514.645133:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829514.645136:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518102144:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787829514.645150:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829514.645156:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518102272:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787829514.645171:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518102656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829514.645185:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829514.645186:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829514.645195:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829514.645197:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829514.645202:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2521 00000100:00080000:2.0:1787829514.645206:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829514.645211:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829514.645217:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff8b2e8801de00 x1874672518102272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829514.645218:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829514.645237:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518102784/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829514.645242:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518102272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829514.645279:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829514.645279:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829514.645292:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829514.645294:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787829514.645297:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518102272:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829514.645300:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829514.645307:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518102400:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787829514.645315:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2521 00000100:00080000:2.0:1787829514.645324:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829514.645330:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829514.645336:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787829514.645339:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829514.645348:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff8b2e8801cb40 x1874672518102400/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829514.645356:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518102400/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829514.645363:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518102912/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829514.645364:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829514.645372:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829514.645377:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518102400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829514.645396:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518102528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829514.645411:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829514.645419:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff8b2e8801f0c0 x1874672518102528/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829514.645426:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518102528/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829514.645433:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829514.645435:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829514.645437:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518102528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829514.645449:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518102656:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829514.645459:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829514.645464:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff8b2e8801c780 x1874672518102656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829514.645475:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518102656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829514.645480:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829514.645481:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829514.645483:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518102656:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829514.645502:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518102784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829514.645516:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829514.645523:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff8b2e8801ed00 x1874672518102784/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829514.645530:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518102784/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829514.645534:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829514.645537:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829514.645538:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518102784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829514.645542:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518102912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829514.645551:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829514.645556:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829514/real 1787829514] req@ffff8b2e8801d680 x1874672518102912/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829514.645563:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518102912/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829539 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829514.645577:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829514.645579:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829514.645580:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518102912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829514.964784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829515.476714:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829515.476786:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829515.668757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829515.668757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829515.988750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829516.500772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829516.500845:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829516.692776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829516.692776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829517.012752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829517.524723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829517.524776:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829517.716759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829517.716769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829518.036805:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829518.548808:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829518.548906:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787829518.740739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829518.740744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829519.060770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829519.572752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829519.572827:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829519.764761:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829519.764765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829519.764767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829519.764776:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829519.764782:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829519.764794:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2526 00000100:00080000:2.0:1787829519.764806:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829519.764814:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829519.764865:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518103040/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829519.764886:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829519.764889:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829519.764891:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829519.764895:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2526 00000100:00080000:2.0:1787829519.764897:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829519.764900:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829519.764910:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829519.764920:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518103168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829519.764928:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829519.764937:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829519.764939:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518103040:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829519.764940:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829519.764946:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2526 00000100:00080000:2.0:1787829519.764953:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829519.764960:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829519.764964:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829519.764973:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518103296/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829519.764980:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829519.764982:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829519.764984:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829519.764987:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2526 00000100:00080000:2.0:1787829519.764990:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829519.764992:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829519.764995:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829519.765006:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518103424/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829519.765013:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829519.765016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829519.765017:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829519.765020:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2526 00000100:00080000:2.0:1787829519.765022:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829519.765024:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829519.765028:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829519.765030:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829519.765042:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518103552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829519.765050:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829519.765053:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829519.765055:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829519.765059:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2526 00000100:00080000:2.0:1787829519.765062:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829519.765064:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829519.765066:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829519.765074:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518103680/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829519.765087:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829519.765089:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff8b2e8801da40 x1874672518103040/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829519.765090:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829519.765093:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829519.765101:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2526 00000100:00080000:2.0:1787829519.765106:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829519.765115:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829519.765125:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829519.765142:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518103808/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829519.765150:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518103040/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.765169:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829519.765174:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829519.765179:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518103040:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829519.765203:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518103168:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829519.765228:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829519.765240:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff8b2e8801fc00 x1874672518103168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829519.765253:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518103168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.765262:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829519.765265:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829519.765269:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518103168:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829519.765275:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518103296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829519.765293:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829519.765304:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff8b2e8801f480 x1874672518103296/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829519.765315:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518103296/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.765323:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829519.765326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829519.765329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518103296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829519.765335:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518103424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829519.765349:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829519.765358:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff8b2e8801cf00 x1874672518103424/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829519.765368:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518103424/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.765375:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829519.765378:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829519.765382:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518103424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829519.765388:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518103552:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829519.765407:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829519.765418:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff8b2e8801c000 x1874672518103552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829519.765429:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518103552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.765437:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829519.765440:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829519.765442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518103552:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829519.765446:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518103680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829519.765472:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829519.765480:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff8b2e8801e580 x1874672518103680/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829519.765486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518103680/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.765491:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829519.765493:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829519.765495:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518103680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829519.765499:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518103808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829519.765508:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829519.765513:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829519/real 1787829519] req@ffff8b2e8801e940 x1874672518103808/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829519.765519:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518103808/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829544 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829519.765524:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829519.765526:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829519.765527:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518103808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829520.084775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829520.596740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829520.596802:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829520.788706:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829520.788722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829521.108741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829521.620744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829521.620772:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829521.812718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829521.812720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829522.132744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787829522.644774:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829522.644824:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787829522.836749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829522.836749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829523.156776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829523.668787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829523.668870:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829523.860754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829523.860756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829524.180775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787829524.692833:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829524.692879:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787829524.884775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829524.884901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829524.884908:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829524.884914:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829524.884922:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829524.884926:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2531 00000100:00080000:0.0:1787829524.884940:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829524.884949:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829524.884994:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518103936/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829524.885016:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829524.885020:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829524.885022:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829524.885026:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2531 00000100:00080000:0.0:1787829524.885031:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829524.885036:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829524.885048:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829524.885048:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518103936:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829524.885062:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518104064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829524.885072:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829524.885075:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829524.885077:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829524.885081:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2531 00000100:00080000:0.0:1787829524.885084:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829524.885087:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829524.885092:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829524.885102:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518104192/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829524.885110:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829524.885113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829524.885115:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829524.885118:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2531 00000100:00080000:0.0:1787829524.885122:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829524.885125:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829524.885130:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829524.885141:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518104320/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829524.885149:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829524.885152:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829524.885154:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829524.885158:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2531 00000100:00080000:0.0:1787829524.885161:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829524.885165:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829524.885170:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829524.885173:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518104064:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829524.885181:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518104448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829524.885190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829524.885194:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829524.885196:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829524.885199:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2531 00000100:00080000:0.0:1787829524.885202:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829524.885203:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518104192:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829524.885206:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829524.885211:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829524.885221:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518104576/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829524.885224:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518104320:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829524.885231:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829524.885234:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829524.885236:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829524.885240:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2531 00000100:00080000:0.0:1787829524.885243:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829524.885247:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829524.885251:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829524.885253:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518104448:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829524.885262:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518104704/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829524.885275:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518104576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829524.885301:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518104704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829525.204785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829525.716768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829525.716873:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829525.908751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829525.908799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829526.228808:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829526.740737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829526.740783:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829526.932756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829526.932795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829527.252768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829527.764761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829527.764815:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829527.956764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829527.956933:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:2.0:1787829527.956973:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787829527.957022:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829527.957043:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829527.957095:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829527.957107:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829527.957118:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829527.957130:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829527.957141:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829527] req@ffff8b2e880421c0 x1874672518104064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787829527.957142:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829527.957158:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829527.957181:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518104064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829527.957197:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829527.957204:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829527.957209:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518104064:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829527.957219:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829527] req@ffff8b2e88040b40 x1874672518104192/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829527.957241:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518104192/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829527.957250:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829527.957254:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829527.957258:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518104192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829527.957266:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829527] req@ffff8b2e880412c0 x1874672518104320/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829527.957278:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518104320/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829527.957287:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829527.957291:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829527.957294:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518104320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829527.957305:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829527] req@ffff8b2e88043840 x1874672518103936/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829527.957321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518103936/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829527.957336:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829527.957340:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829527.957345:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518103936:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829527.957385:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829527] req@ffff8b2e88040780 x1874672518104704/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829527.957416:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518104704/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829527.957425:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829527.957429:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829527.957432:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518104704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829527.957440:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829527] req@ffff8b2e88042580 x1874672518104448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829527.957453:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518104448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829527.957461:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829527.957465:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829527.957469:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518104448:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829527.957478:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829524/real 1787829527] req@ffff8b2e88042d00 x1874672518104576/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829527.957492:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518104576/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829549 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829527.957501:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829527.957505:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829527.957509:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518104576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829528.276762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829528.788751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829528.788806:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829528.980748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829528.980787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829529.300790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829529.812799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829529.812840:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829530.004745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829530.004757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829530.004813:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829530.004820:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829530.004823:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829530.004832:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2536 00000100:00080000:1.0:1787829530.004842:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829530.004849:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829530.004880:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518104832/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829530.004910:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829530.004914:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829530.004916:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829530.004920:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2536 00000100:00080000:1.0:1787829530.004923:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829530.004927:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829530.004937:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829530.004955:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518104832:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829530.004957:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518104960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829530.004970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829530.004974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829530.004976:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829530.004979:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2536 00000100:00080000:1.0:1787829530.004983:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829530.004987:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829530.004992:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829530.005004:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518105088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829530.005014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829530.005019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787829530.005020:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829530.005022:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829530.005028:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2536 00000100:00080000:1.0:1787829530.005038:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829530.005043:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829530.005048:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829530.005061:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518105216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829530.005064:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829530/real 1787829530] req@ffff8b2e88077840 x1874672518104832/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829530.005080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829530.005084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829530.005086:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829530.005091:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2536 00000100:00080000:1.0:1787829530.005095:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829530.005098:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518104832/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829530.005102:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829530.005110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829530.005114:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829530.005121:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829530.005125:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518105344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829530.005128:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518104832:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829530.005143:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829530.005146:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829530.005149:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829530.005154:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2536 00000100:00080000:1.0:1787829530.005157:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829530.005159:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518104960:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829530.005165:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829530.005172:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829530.005185:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518105472/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829530.005192:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829530.005208:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829530.005209:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829530/real 1787829530] req@ffff8b2e88076580 x1874672518104960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829530.005216:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829530.005220:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829530.005228:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2536 00000100:00080000:1.0:1787829530.005235:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829530.005241:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829530.005247:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829530.005249:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518104960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829530.005261:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829530.005262:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518105600/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829530.005269:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829530.005277:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518104960:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829530.005287:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518105088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829530.005309:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829530.005331:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829530/real 1787829530] req@ffff8b2e88076d00 x1874672518105088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829530.005342:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518105088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829530.005350:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829530.005353:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829530.005356:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518105088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829530.005362:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518105216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829530.005389:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829530.005401:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829530/real 1787829530] req@ffff8b2e880770c0 x1874672518105216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829530.005411:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518105216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829530.005422:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829530.005425:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829530.005428:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518105216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829530.005434:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518105344:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829530.005451:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829530.005461:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829530/real 1787829530] req@ffff8b2e88074000 x1874672518105344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829530.005472:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518105344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829530.005481:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829530.005484:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829530.005487:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518105344:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829530.005503:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518105472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829530.005524:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829530.005534:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829530/real 1787829530] req@ffff8b2e880761c0 x1874672518105472/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829530.005545:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518105472/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829530.005552:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829530.005555:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829530.005558:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518105472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829530.005563:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518105600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829530.005680:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829530.005692:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829530/real 1787829530] req@ffff8b2e88075e00 x1874672518105600/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829530.005702:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518105600/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829555 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829530.005709:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829530.005712:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829530.005715:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518105600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829530.324781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829530.836736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829530.836783:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829531.028729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829531.028761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829531.348763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829531.860724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829531.860786:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829532.052754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829532.052754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829532.372769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829532.884758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829532.884792:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829533.076750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829533.076771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829533.396770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829533.908760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829533.908823:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829534.100774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829534.100775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829534.420733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829534.932734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829534.932809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829535.124772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829535.124780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829535.124781:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829535.124786:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829535.124792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829535.124804:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2541 00000100:00080000:1.0:1787829535.124817:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829535.124825:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829535.124870:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518105728/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829535.124890:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829535.124893:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829535.124895:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829535.124898:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2541 00000100:00080000:1.0:1787829535.124900:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829535.124903:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829535.124913:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829535.124928:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518105728:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829535.124931:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518105856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829535.124946:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829535.124950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829535.124952:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829535.124956:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2541 00000100:00080000:1.0:1787829535.124959:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829535.124963:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829535.124969:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829535.124982:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518105984/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829535.124992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829535.124995:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829535.124998:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829535.125002:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2541 00000100:00080000:1.0:1787829535.125007:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829535.125010:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829535.125015:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829535.125025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518106112/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829535.125030:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829535.125041:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829535.125045:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829535.125048:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829535.125053:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2541 00000100:00080000:1.0:1787829535.125062:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829535.125067:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829535.125072:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829535.125086:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518106240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829535.125096:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829535.125100:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829535.125102:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829535/real 1787829535] req@ffff8b2e88074b40 x1874672518105728/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829535.125103:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829535.125113:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2541 00000100:00080000:1.0:1787829535.125120:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829535.125127:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829535.125133:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829535.125138:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518105728/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829535.125148:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518106368/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829535.125158:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829535.125159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829535.125166:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829535.125168:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829535.125170:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829535.125174:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2541 00000100:00080000:1.0:1787829535.125177:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829535.125181:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518105728:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829535.125183:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829535.125189:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829535.125200:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518106496/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829535.125223:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518105856:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829535.125253:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829535.125268:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829535/real 1787829535] req@ffff8b2e88075e00 x1874672518105856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829535.125283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518105856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829535.125293:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829535.125297:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829535.125301:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518105856:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829535.125308:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518105984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829535.125329:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829535.125341:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829535/real 1787829535] req@ffff8b2e880761c0 x1874672518105984/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829535.125372:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518105984/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829535.125382:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829535.125386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829535.125390:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518105984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829535.125396:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518106112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829535.125419:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829535.125433:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829535/real 1787829535] req@ffff8b2e88074000 x1874672518106112/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829535.125447:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518106112/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829535.125456:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829535.125460:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829535.125464:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518106112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829535.125470:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518106240:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829535.125489:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829535.125501:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829535/real 1787829535] req@ffff8b2e880770c0 x1874672518106240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829535.125513:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518106240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829535.125522:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829535.125525:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829535.125528:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518106240:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829535.125535:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518106368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829535.125554:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829535.125565:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829535/real 1787829535] req@ffff8b2e88076d00 x1874672518106368/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829535.125579:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518106368/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829535.125594:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829535.125597:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829535.125600:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518106368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829535.125617:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518106496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829535.125639:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829535.125682:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829535/real 1787829535] req@ffff8b2e88076580 x1874672518106496/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829535.125696:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518106496/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829560 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829535.125705:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829535.125708:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829535.125712:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518106496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829535.444727:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829535.956777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829535.956807:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829536.148750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829536.148755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829536.468766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829536.980755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829536.980890:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829537.172788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829537.172790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829537.492710:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829538.004751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829538.004785:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829538.196728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829538.196743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829538.516772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829539.028731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829539.028849:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829539.220769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829539.220773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829539.540762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829540.052740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829540.052801:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829540.244774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829540.244774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829540.244943:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829540.244951:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829540.244956:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829540.244967:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2546 00000100:00080000:1.0:1787829540.244980:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829540.244989:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829540.245030:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518106624/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829540.245060:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829540.245064:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829540.245065:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829540.245070:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2546 00000100:00080000:1.0:1787829540.245074:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829540.245080:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829540.245091:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829540.245100:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518106624:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829540.245109:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518106752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829540.245124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829540.245128:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829540.245130:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829540.245136:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2546 00000100:00080000:1.0:1787829540.245141:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829540.245145:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829540.245152:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829540.245168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518106880/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829540.245172:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829540.245185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829540.245190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829540.245193:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829540.245199:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2546 00000100:00080000:1.0:1787829540.245203:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829540.245207:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829540.245214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829540.245218:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff8b2e88077840 x1874672518106624/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829540.245232:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518107008/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829540.245243:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518106624/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829540.245248:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829540.245259:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829540.245262:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829540.245263:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829540.245268:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829540.245275:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2546 00000100:00100000:3.0:1787829540.245276:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518106624:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829540.245285:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829540.245293:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829540.245299:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829540.245304:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518106752:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829540.245332:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518107136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829540.245345:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829540.245348:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829540.245354:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829540.245357:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829540.245363:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff8b2e88074780 x1874672518106752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829540.245364:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2546 00000100:00080000:1.0:1787829540.245372:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829540.245381:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829540.245389:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829540.245390:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518106752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.245405:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787829540.245410:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518107264/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829540.245413:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829540.245423:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518106752:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829540.245437:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829540.245438:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518106880:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829540.245447:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829540.245451:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829540.245458:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2546 00000100:00080000:1.0:1787829540.245463:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829540.245469:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829540.245472:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787829540.245479:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829540.245489:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff8b2e88077c00 x1874672518106880/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829540.245498:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518107392/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829540.245509:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518106880/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.245528:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829540.245532:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829540.245536:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518106880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829540.245544:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518107008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829540.245562:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829540.245573:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff8b2e88075680 x1874672518107008/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829540.245586:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518107008/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.245594:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829540.245598:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829540.245601:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518107008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829540.245617:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518107136:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829540.245635:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829540.245664:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff8b2e88075a40 x1874672518107136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829540.245681:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518107136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.245691:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829540.245695:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829540.245699:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518107136:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829540.245705:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518107264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829540.245728:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829540.245740:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff8b2e88077480 x1874672518107264/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829540.245754:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518107264/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.245774:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829540.245777:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829540.245782:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518107264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829540.245787:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518107392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829540.245804:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829540.245812:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829540/real 1787829540] req@ffff8b2e88076940 x1874672518107392/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829540.245820:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518107392/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829565 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829540.245826:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829540.245828:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829540.245830:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518107392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829540.564777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829541.076776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829541.076921:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829541.268781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829541.268794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829541.588728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829542.100777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829542.100821:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829542.292797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829542.292801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829542.612780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829543.124780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829543.124847:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829543.316800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829543.316807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829543.636749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829544.148737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829544.148812:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829544.340748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829544.340748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829544.660781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829545.172751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829545.172803:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829545.364778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829545.364778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829545.364786:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.364795:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829545.364801:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.364813:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2551 00000100:00080000:2.0:1787829545.364830:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829545.364840:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829545.364891:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518107520/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829545.364922:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.364927:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829545.364929:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.364936:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2551 00000100:00080000:2.0:1787829545.364941:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829545.364947:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829545.364961:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829545.364969:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518107520:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829545.364985:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518107648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829545.365010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.365014:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829545.365017:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.365023:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2551 00000100:00080000:2.0:1787829545.365033:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829545.365039:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829545.365046:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787829545.365048:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829545.365078:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518107776/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829545.365092:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff8b2e8801e940 x1874672518107520/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829545.365093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.365112:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829545.365115:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.365125:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2551 00000100:00080000:2.0:1787829545.365132:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829545.365137:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518107520/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829545.365141:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829545.365160:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829545.365161:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829545.365172:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829545.365177:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518107520:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829545.365187:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518107904/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829545.365202:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518107648:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829545.365203:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.365218:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829545.365222:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.365231:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2551 00000800:00000100:3.0:1787829545.365237:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829545.365239:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829545.365249:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829545.365253:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff8b2e8801e580 x1874672518107648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829545.365258:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829545.365284:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518107648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829545.365296:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787829545.365299:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518108032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829545.365301:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829545.365308:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518107648:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829545.365328:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518107776:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787829545.365336:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.365342:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829545.365345:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.365354:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2551 00000800:00000100:3.0:1787829545.365355:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829545.365366:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829545.365372:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff8b2e8801c000 x1874672518107776/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829545.365384:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829545.365385:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518107776/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829545.365399:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829545.365401:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829545.365408:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829545.365412:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518107776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787829545.365424:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518108160/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829545.365428:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518107904:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787829545.365453:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829545.365461:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787829545.365461:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829545.365470:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829545.365479:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2551 00000100:00080000:2.0:1787829545.365485:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829545.365491:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829545.365497:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829545.365499:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff8b2e8801cf00 x1874672518107904/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829545.365517:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518108288/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829545.365522:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518107904/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829545.365594:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829545.365602:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829545.365608:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518107904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829545.365618:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518108032:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829545.365639:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829545.365669:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff8b2e8801f480 x1874672518108032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829545.365679:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518108032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829545.365690:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829545.365692:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829545.365694:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518108032:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829545.365706:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518108160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829545.365719:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829545.365726:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff8b2e8801fc00 x1874672518108160/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829545.365737:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518108160/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829545.365743:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829545.365746:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829545.365747:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518108160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829545.365752:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518108288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829545.365764:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829545.365770:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829545/real 1787829545] req@ffff8b2e8801da40 x1874672518108288/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829545.365777:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518108288/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829570 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829545.365782:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829545.365784:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829545.365787:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518108288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829545.684730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829546.196795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829546.196824:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829546.388730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829546.388731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829546.708772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829547.220802:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829547.220869:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829547.412768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829547.412768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829547.732712:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829548.244760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829548.244817:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829548.436722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829548.436724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829548.756760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829549.268784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829549.268824:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829549.460789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829549.460796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829549.780786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829550.292825:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829550.292872:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829550.484745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829550.484752:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829550.484754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829550.484765:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829550.484771:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829550.484784:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2556 00000100:00080000:2.0:1787829550.484799:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829550.484806:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829550.484850:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518108416/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829550.484878:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829550.484881:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829550.484883:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829550.484886:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2556 00000100:00080000:2.0:1787829550.484890:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829550.484894:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829550.484904:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829550.484918:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518108416:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829550.484926:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518108544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829550.484938:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829550.484941:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829550.484943:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829550.484946:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2556 00000100:00080000:2.0:1787829550.484950:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829550.484954:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829550.484959:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829550.484968:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518108672/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829550.484977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829550.484981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829550.484982:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829550.484987:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2556 00000800:00000100:3.0:1787829550.484989:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829550.484996:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829550.485000:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829550.485004:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829550.485015:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518108800/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829550.485022:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829550.485025:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829550.485026:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829550.485029:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2556 00000100:00080000:2.0:1787829550.485031:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829550.485034:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829550.485038:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829550.485052:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518108928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829550.485054:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff8b2e8801d680 x1874672518108416/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829550.485091:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518108416/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829550.485100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829550.485117:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829550.485119:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829550.485121:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829550.485128:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2556 00000100:00080000:3.0:1787829550.485133:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829550.485140:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829550.485147:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518108416:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829550.485152:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829550.485163:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829550.485174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518109056/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829550.485181:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518108544:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829550.485196:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829550.485200:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829550.485202:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829550.485210:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2556 00000100:00080000:2.0:1787829550.485213:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829550.485217:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829550.485219:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829550.485228:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829550.485252:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518109184/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829550.485256:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff8b2e8801ed00 x1874672518108544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829550.485304:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518108544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829550.485311:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829550.485316:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829550.485318:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518108544:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829550.485326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518108672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829550.485344:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829550.485352:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff8b2e8801c780 x1874672518108672/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829550.485363:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518108672/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829550.485372:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829550.485375:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829550.485378:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518108672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829550.485386:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518108800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829550.485407:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829550.485418:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff8b2e8801f0c0 x1874672518108800/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829550.485429:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518108800/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829550.485436:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829550.485440:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829550.485443:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518108800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829550.485449:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518108928:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829550.485467:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829550.485476:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff8b2e8801cb40 x1874672518108928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829550.485488:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518108928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829550.485496:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829550.485499:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829550.485502:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518108928:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829550.485520:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518109056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829550.485539:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829550.485549:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff8b2e8801de00 x1874672518109056/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829550.485560:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518109056/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829550.485567:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829550.485571:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829550.485574:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518109056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829550.485580:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518109184:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829550.485595:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829550.485606:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829550/real 1787829550] req@ffff8b2e8801c3c0 x1874672518109184/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829550.485616:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518109184/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829575 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829550.485630:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829550.485633:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829550.485636:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518109184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829550.804731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829551.316777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829551.316842:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829551.508743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829551.508743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829551.828808:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787829552.340764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829552.340834:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829552.532738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829552.532738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829552.852732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829553.364815:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829553.364867:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829553.556747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829553.556747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829553.876731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829554.388738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829554.388866:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829554.580787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829554.580793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829554.900776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829555.412742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829555.412792:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829555.604768:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.604780:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829555.604786:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.604802:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2562 00000100:00100000:1.0:1787829555.604808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829555.604818:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.604830:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829555.604881:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518109312/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829555.604921:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.604925:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829555.604927:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.604933:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2562 00000100:00080000:0.0:1787829555.604937:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.604942:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829555.604955:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829555.604956:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518109312:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829555.604976:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518109440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829555.604989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.604993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829555.604995:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.605008:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2562 00000100:00080000:0.0:1787829555.605014:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.605018:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829555.605024:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829555.605037:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518109568/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829555.605044:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829555.605055:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.605060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829555.605062:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.605068:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2562 00000100:00080000:0.0:1787829555.605083:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.605089:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829555.605095:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829555.605108:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff8b2e88041e00 x1874672518109312/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829555.605112:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518109696/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829555.605141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.605145:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829555.605147:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829555.605150:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518109312/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829555.605156:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2562 00000100:00080000:0.0:1787829555.605165:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.605172:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829555.605173:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829555.605181:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829555.605184:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829555.605191:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518109312:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829555.605199:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518109824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829555.605228:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829555.605232:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829555.605235:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829555.605238:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518109440:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829555.605244:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2562 00000100:00080000:0.0:1787829555.605251:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829555.605257:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829555.605263:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829555.605279:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829555.605281:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518109952/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829555.605300:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829555.605300:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff8b2e88042d00 x1874672518109440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829555.605309:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829555.605314:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829555.605325:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2562 00000100:00080000:0.0:1787829555.605330:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829555.605331:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518109440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829555.605336:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829555.605345:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829555.605360:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518110080/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829555.605364:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829555.605373:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829555.605378:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518109440:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829555.605386:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829555.605390:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518109568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829555.605418:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829555.605431:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff8b2e88042580 x1874672518109568/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829555.605444:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518109568/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829555.605454:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829555.605457:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829555.605461:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518109568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829555.605468:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518109696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829555.605490:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829555.605501:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff8b2e88040780 x1874672518109696/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829555.605513:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518109696/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829555.605522:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829555.605526:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829555.605529:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518109696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829555.605536:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518109824:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829555.605559:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829555.605572:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff8b2e88043840 x1874672518109824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829555.605585:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518109824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829555.605595:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829555.605598:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829555.605601:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518109824:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829555.605620:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518109952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829555.605642:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829555.605681:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff8b2e880412c0 x1874672518109952/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829555.605694:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518109952/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829555.605703:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829555.605707:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829555.605710:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518109952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829555.605717:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518110080:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829555.605737:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829555.605750:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829555/real 1787829555] req@ffff8b2e88040b40 x1874672518110080/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829555.605769:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518110080/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829580 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829555.605778:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829555.605781:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829555.605785:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518110080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829555.924785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829556.436740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829556.436769:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829556.628757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829556.628758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829556.948767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829557.460764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829557.460834:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829557.652735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829557.652754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829557.972769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829558.484787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829558.484825:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829558.676768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829558.676778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829558.996760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829559.508745:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829559.508802:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829559.700759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829559.700760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829560.020765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829560.532725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829560.532856:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829560.724770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829560.724794:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829560.724798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829560.724803:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829560.724810:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.724823:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2567 00000100:00080000:2.0:1787829560.724835:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.724844:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829560.724886:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518110208/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829560.724913:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829560.724917:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829560.724920:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.724924:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2567 00000100:00080000:2.0:1787829560.724928:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.724933:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829560.724943:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829560.724952:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518110208:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829560.724957:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518110336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829560.724970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829560.724973:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829560.724976:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.724980:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2567 00000100:00080000:2.0:1787829560.724983:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.724988:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829560.724994:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829560.725003:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518110464/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829560.725010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829560.725012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829560.725013:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.725016:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2567 00000100:00080000:2.0:1787829560.725019:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.725021:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829560.725024:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829560.725028:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829560.725035:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518110592/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829560.725042:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829560.725045:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829560.725047:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.725050:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2567 00000100:00080000:2.0:1787829560.725052:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.725054:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829560.725057:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829560.725065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518110720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829560.725071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829560.725074:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff8b2e8801c3c0 x1874672518110208/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829560.725075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829560.725078:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829560.725084:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2567 00000100:00080000:2.0:1787829560.725089:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.725094:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829560.725098:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829560.725103:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518110208/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829560.725107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518110848/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829560.725132:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829560.725133:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829560.725136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829560.725139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829560.725140:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829560.725147:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2567 00000100:00100000:3.0:1787829560.725148:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518110208:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829560.725151:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829560.725157:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829560.725160:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829560.725182:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518110976/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829560.725188:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518110336:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829560.725212:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829560.725228:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff8b2e8801de00 x1874672518110336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829560.725237:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518110336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829560.725244:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829560.725246:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829560.725249:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518110336:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829560.725254:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518110464:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829560.725267:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829560.725273:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff8b2e8801cb40 x1874672518110464/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829560.725281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518110464/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829560.725286:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829560.725288:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829560.725290:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518110464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829560.725299:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518110592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829560.725310:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829560.725317:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff8b2e8801f0c0 x1874672518110592/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829560.725324:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518110592/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829560.725329:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829560.725331:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829560.725334:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518110592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829560.725338:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518110720:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829560.725348:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829560.725357:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff8b2e8801c780 x1874672518110720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829560.725378:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518110720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829560.725387:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829560.725391:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829560.725395:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518110720:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829560.725401:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518110848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829560.725419:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829560.725427:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff8b2e8801ed00 x1874672518110848/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829560.725435:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518110848/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829560.725440:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829560.725442:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829560.725444:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518110848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829560.725458:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518110976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829560.725469:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829560.725475:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829560/real 1787829560] req@ffff8b2e8801d680 x1874672518110976/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829560.725483:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518110976/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829585 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829560.725488:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829560.725490:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829560.725492:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518110976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829561.044717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829561.556740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829561.556808:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829561.748739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829561.748739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829562.068726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829562.580759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829562.580780:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829562.772767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829562.772767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829563.092717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829563.604726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829563.604851:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829563.796751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829563.796751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829564.116759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787829564.628838:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829564.628867:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787829564.820768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829564.820768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829565.140719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787829565.652759:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829565.652777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787829565.844732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829565.844737:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829565.844744:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829565.844749:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829565.844757:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2572 00000100:00100000:3.0:1787829565.844758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829565.844768:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829565.844775:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829565.844808:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518111104/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829565.844829:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829565.844832:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829565.844833:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829565.844836:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2572 00000100:00080000:2.0:1787829565.844839:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829565.844842:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829565.844850:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829565.844858:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518111232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829565.844866:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829565.844869:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829565.844872:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829565.844873:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518111104:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829565.844877:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2572 00000100:00080000:2.0:1787829565.844881:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829565.844885:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829565.844888:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829565.844897:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518111360/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829565.844905:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829565.844907:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829565.844908:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829565.844911:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2572 00000100:00080000:2.0:1787829565.844913:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829565.844915:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829565.844918:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829565.844925:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518111488/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829565.844930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829565.844933:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829565.844934:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829565.844937:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2572 00000100:00080000:2.0:1787829565.844940:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829565.844942:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829565.844945:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829565.844949:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829565.844955:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518111616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829565.844962:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829565.844964:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829565.844965:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829565.844968:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2572 00000100:00080000:2.0:1787829565.844972:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829565.844974:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829565.844978:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829565.844985:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518111744/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829565.844990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829565.844992:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829565.844994:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829565.844996:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2572 00000100:00080000:2.0:1787829565.844999:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829565.845002:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829565.845005:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829565.845007:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829565] req@ffff8b2e8801da40 x1874672518111104/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829565.845015:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518111872/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829565.845045:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518111104/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829565.845057:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829565.845062:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829565.845066:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518111104:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829565.845091:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518111232:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829565.845122:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829565.845155:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829565] req@ffff8b2e8801fc00 x1874672518111232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829565.845168:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518111232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829565.845178:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829565.845181:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829565.845185:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518111232:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829565.845192:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518111360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829565.845215:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829565.845228:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829565] req@ffff8b2e8801f480 x1874672518111360/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829565.845242:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518111360/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829565.845251:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829565.845255:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829565.845259:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518111360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829565.845266:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518111488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829565.845286:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829565.845299:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829565] req@ffff8b2e8801cf00 x1874672518111488/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829565.845320:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518111488/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829565.845340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829565.845344:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829565.845348:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518111488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829565.845355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518111616:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829565.845377:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829565.845391:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829565] req@ffff8b2e8801c000 x1874672518111616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829565.845404:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518111616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829565.845414:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829565.845418:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829565.845422:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518111616:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829565.845429:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518111744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829565.845450:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829565.845463:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829565] req@ffff8b2e8801e580 x1874672518111744/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829565.845475:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518111744/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829565.845492:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829565.845495:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829565.845499:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518111744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829565.845506:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518111872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829565.845526:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829565.845539:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829565/real 1787829565] req@ffff8b2e8801e940 x1874672518111872/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829565.845552:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518111872/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829590 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829565.845560:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829565.845564:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829565.845567:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518111872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829566.164754:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787829566.676898:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829566.676933:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787829566.868758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829566.868759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829567.188729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829567.700736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829567.700801:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829567.892728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829567.892728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829568.212750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829568.724776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829568.724818:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829568.916790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829568.916790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829569.236810:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829569.748725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829569.748831:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829569.940825:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829569.940825:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829570.260711:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829570.772764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829570.772839:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829570.964768:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829570.964771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829570.964772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829570.964781:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829570.964787:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829570.964798:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2577 00000100:00080000:2.0:1787829570.964809:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829570.964818:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829570.964863:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518112000/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829570.964899:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829570.964903:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829570.964905:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829570.964909:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2577 00000100:00080000:2.0:1787829570.964913:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829570.964918:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829570.964929:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829570.964934:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518112000:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829570.964946:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518112128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829570.964959:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829570.964963:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829570.964965:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829570.964970:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2577 00000100:00080000:2.0:1787829570.964975:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829570.964979:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829570.964985:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829570.964988:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829570.965000:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518112256/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829570.965011:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829570.965014:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829570.965016:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829570.965020:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2577 00000100:00080000:2.0:1787829570.965024:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829570.965028:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829570.965030:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff8b2e8801e940 x1874672518112000/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829570.965035:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829570.965055:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518112384/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829570.965055:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518112000/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829570.965083:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829570.965085:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829570.965090:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829570.965091:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829570.965107:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829570.965111:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518112000:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829570.965115:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2577 00000100:00080000:2.0:1787829570.965121:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829570.965124:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829570.965129:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829570.965138:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518112128:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787829570.965141:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518112512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829570.965151:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829570.965153:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829570.965155:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829570.965158:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2577 00000100:00080000:2.0:1787829570.965161:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829570.965165:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829570.965167:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829570.965172:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829570.965181:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff8b2e8801e580 x1874672518112128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829570.965183:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518112640/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829570.965209:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829570.965211:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518112128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829570.965218:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829570.965223:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829570.965225:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829570.965232:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2577 00000100:00080000:3.0:1787829570.965232:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829570.965240:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518112128:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829570.965241:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829570.965250:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829570.965253:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518112256:192.168.203.101@tcp:8:kworker.0 00000008:00100000:2.0:1787829570.965258:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829570.965276:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518112768/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829570.965286:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829570.965302:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff8b2e8801c000 x1874672518112256/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829570.965312:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518112256/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829570.965322:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829570.965324:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829570.965327:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518112256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829570.965331:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518112384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829570.965344:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829570.965350:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff8b2e8801cf00 x1874672518112384/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829570.965357:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518112384/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829570.965363:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829570.965364:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829570.965366:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518112384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829570.965377:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518112512:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829570.965388:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829570.965395:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff8b2e8801f480 x1874672518112512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829570.965402:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518112512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829570.965408:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829570.965409:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829570.965412:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518112512:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829570.965415:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518112640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829570.965425:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829570.965430:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff8b2e8801fc00 x1874672518112640/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829570.965437:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518112640/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829570.965443:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829570.965444:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829570.965446:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518112640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829570.965450:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518112768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829570.965459:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829570.965464:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829570/real 1787829570] req@ffff8b2e8801da40 x1874672518112768/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829570.965471:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518112768/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829595 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829570.965476:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829570.965478:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829570.965480:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518112768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829571.284759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829571.796742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829571.796795:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829571.988742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829571.988742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829572.308722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829572.820750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829572.820812:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829573.012729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829573.012734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829573.332781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829573.844766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829573.844848:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829574.036782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829574.036826:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829574.356719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829574.868773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829574.868793:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829575.060725:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829575.060776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829575.380771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829575.892703:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829575.892817:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829576.084731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829576.084732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829576.084778:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829576.084783:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829576.084787:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829576.084795:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2582 00000100:00080000:3.0:1787829576.084805:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829576.084810:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829576.084842:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518112896/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829576.084866:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829576.084869:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829576.084870:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829576.084873:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2582 00000100:00080000:3.0:1787829576.084876:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829576.084879:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829576.084888:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829576.084896:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518112896:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829576.084899:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518113024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829576.084909:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829576.084911:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829576.084912:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829576.084916:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2582 00000100:00080000:3.0:1787829576.084918:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829576.084921:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829576.084925:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829576.084933:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518113152/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829576.084939:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829576.084941:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829576.084943:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829576.084945:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2582 00000100:00080000:3.0:1787829576.084947:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829576.084949:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829576.084952:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829576.084955:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829576.084962:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518113280/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829576.084969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829576.084971:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829576.084972:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829576.084975:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2582 00000100:00080000:3.0:1787829576.084977:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829576.084979:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829576.084982:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829576.084990:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518113408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829576.084996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829576.084999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829576.085001:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829576.085003:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829576/real 1787829576] req@ffff8b2e88039680 x1874672518112896/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829576.085004:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2582 00000100:00080000:3.0:1787829576.085009:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829576.085013:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829576.085019:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829576.085029:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518113536/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829576.085036:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829576.085038:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518112896/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829576.085039:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829576.085042:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829576.085049:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2582 00000100:00080000:0.0:1787829576.085074:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829576.085077:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829576.085082:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829576.085086:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829576.085092:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518112896:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787829576.085095:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829576.085109:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518113664/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829576.085124:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518113024:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829576.085154:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829576.085169:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829576/real 1787829576] req@ffff8b2e8803b480 x1874672518113024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829576.085183:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518113024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829576.085198:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829576.085201:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829576.085206:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518113024:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829576.085212:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518113152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829576.085235:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829576.085249:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829576/real 1787829576] req@ffff8b2e88039e00 x1874672518113152/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829576.085262:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518113152/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829576.085271:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829576.085274:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829576.085278:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518113152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829576.085284:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518113280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829576.085306:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829576.085318:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829576/real 1787829576] req@ffff8b2e88039a40 x1874672518113280/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829576.085329:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518113280/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829576.085338:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829576.085341:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829576.085345:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518113280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829576.085351:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518113408:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829576.085387:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829576.085399:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829576/real 1787829576] req@ffff8b2e880392c0 x1874672518113408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829576.085412:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518113408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829576.085421:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829576.085425:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829576.085428:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518113408:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829576.085434:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518113536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829576.085456:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829576.085468:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829576/real 1787829576] req@ffff8b2e880383c0 x1874672518113536/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829576.085480:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518113536/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829576.085488:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829576.085492:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829576.085495:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518113536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829576.085501:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518113664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829576.085522:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829576.085533:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829576/real 1787829576] req@ffff8b2e8803a580 x1874672518113664/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829576.085545:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518113664/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829601 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829576.085554:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829576.085557:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829576.085560:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518113664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829576.404718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829576.916762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829576.916864:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829577.108763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829577.108772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829577.428731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829577.940746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829577.940784:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829578.132816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829578.132817:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829578.452767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829578.964732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829578.964792:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829579.156729:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829579.156733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829579.476779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829579.988737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829579.988822:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829580.180735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829580.180736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829580.500746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829581.012753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829581.012761:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829581.204745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829581.204797:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829581.204799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829581.204805:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829581.204810:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829581.204819:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2587 00000100:00080000:0.0:1787829581.204829:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829581.204838:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829581.204880:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518113792/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829581.204908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829581.204912:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829581.204914:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829581.204919:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2587 00000100:00080000:0.0:1787829581.204931:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829581.204937:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829581.204947:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829581.204952:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518113792:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829581.204963:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518113920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829581.204978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829581.204981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829581.204983:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829581.204987:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2587 00000100:00080000:0.0:1787829581.204992:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829581.204996:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829581.205001:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829581.205015:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518114048/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829581.205040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829581.205043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829581.205045:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829581.205048:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2587 00000100:00080000:0.0:1787829581.205052:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829581.205056:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829581.205066:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829581.205067:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829581.205084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518114176/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829581.205099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829581.205102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829581.205105:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829581.205109:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2587 00000100:00080000:0.0:1787829581.205115:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829581.205121:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829581.205126:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829581/real 1787829581] req@ffff8b2e880421c0 x1874672518113792/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829581.205130:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829581.205151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518114304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829581.205187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829581.205190:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829581.205194:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518113792/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829581.205195:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829581.205212:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2587 00000100:00080000:3.0:1787829581.205217:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829581.205219:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829581.205223:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829581.205233:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829581.205233:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518113792:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829581.205252:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829581.205270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518114432/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829581.205273:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518113920:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829581.205304:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829581.205307:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829581.205309:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829581.205313:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2587 00000100:00080000:0.0:1787829581.205317:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829581.205322:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829581.205326:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829581.205330:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829581.205346:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518114560/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829581.205358:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829581/real 1787829581] req@ffff8b2e88040f00 x1874672518113920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829581.205379:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518113920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829581.205389:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829581.205394:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829581.205399:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518113920:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829581.205408:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518114048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829581.205434:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829581.205452:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829581/real 1787829581] req@ffff8b2e88041a40 x1874672518114048/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829581.205460:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518114048/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829581.205471:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829581.205473:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829581.205475:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518114048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829581.205480:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518114176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829581.205492:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829581.205498:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829581/real 1787829581] req@ffff8b2e88040000 x1874672518114176/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829581.205514:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518114176/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829581.205521:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829581.205523:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829581.205525:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518114176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829581.205529:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518114304:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829581.205558:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829581.205572:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829581/real 1787829581] req@ffff8b2e880430c0 x1874672518114304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829581.205586:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518114304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829581.205593:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829581.205596:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829581.205599:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518114304:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829581.205614:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518114432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829581.205630:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829581.205638:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829581/real 1787829581] req@ffff8b2e88043480 x1874672518114432/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829581.205676:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518114432/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829581.205690:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829581.205695:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829581.205699:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518114432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829581.205707:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518114560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829581.205730:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829581.205740:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829581/real 1787829581] req@ffff8b2e88042940 x1874672518114560/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829581.205748:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518114560/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829606 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829581.205754:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829581.205756:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829581.205758:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518114560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829581.524752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829582.036776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829582.036847:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829582.228783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829582.228788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829582.548756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829583.060724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829583.060838:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829583.252779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829583.252779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829583.572791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829584.084747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829584.084789:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829584.276761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829584.276761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829584.596718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829585.108746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829585.108813:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829585.300771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829585.300788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829585.620746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829586.132772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829586.132863:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829586.324755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829586.324755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829586.324770:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829586.324780:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829586.324793:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829586.324807:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2592 00000100:00080000:0.0:1787829586.324821:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829586.324831:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829586.324882:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518114688/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829586.324906:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829586.324910:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829586.324913:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829586.324919:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2592 00000100:00080000:0.0:1787829586.324924:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829586.324931:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829586.324940:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518114688:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829586.324943:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829586.324965:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518114816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829586.324982:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829586.325025:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829586.325051:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829586.325055:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829586.325061:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2592 00000100:00080000:0.0:1787829586.325066:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829586.325072:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829586.325079:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829586.325079:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff8b2e880403c0 x1874672518114688/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829586.325101:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518114944/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829586.325107:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518114688/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829586.325127:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829586.325131:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829586.325138:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829586.325145:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518114688:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829586.325148:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829586.325153:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829586.325158:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2592 00000100:00080000:0.0:1787829586.325163:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829586.325166:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829586.325171:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829586.325179:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518114816:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829586.325185:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518115072/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829586.325204:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829586.325207:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829586.325208:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829586.325212:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2592 00000100:00080000:0.0:1787829586.325217:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829586.325220:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829586.325224:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829586.325227:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829586.325238:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518115200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829586.325246:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829586.325250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff8b2e88042940 x1874672518114816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829586.325255:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829586.325259:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829586.325267:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2592 00000100:00080000:0.0:1787829586.325271:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829586.325274:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829586.325278:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518114816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829586.325279:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829586.325294:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829586.325299:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518115328/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829586.325303:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829586.325312:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518114816:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829586.325318:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829586.325321:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829586.325323:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829586.325325:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518114944:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829586.325329:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2592 00000100:00080000:0.0:1787829586.325336:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829586.325338:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829586.325342:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829586.325352:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518115456/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829586.325366:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829586.325386:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff8b2e88043480 x1874672518114944/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829586.325401:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518114944/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829586.325411:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829586.325414:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829586.325418:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518114944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829586.325432:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518115072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829586.325452:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829586.325463:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff8b2e880430c0 x1874672518115072/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829586.325476:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518115072/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829586.325485:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829586.325489:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829586.325493:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518115072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829586.325500:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518115200:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829586.325521:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829586.325533:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff8b2e88040000 x1874672518115200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829586.325545:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518115200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829586.325554:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829586.325558:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829586.325561:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518115200:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829586.325568:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518115328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829586.325589:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829586.325601:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff8b2e88041a40 x1874672518115328/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829586.325614:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518115328/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829586.325623:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829586.325626:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829586.325629:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518115328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829586.325635:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518115456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829586.325680:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829586.325693:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829586/real 1787829586] req@ffff8b2e88040f00 x1874672518115456/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829586.325707:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518115456/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829611 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829586.325716:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829586.325719:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829586.325724:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518115456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829586.644732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829587.156780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829587.156805:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829587.348756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829587.348756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829587.668772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829588.180776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829588.180883:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829588.372761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829588.372761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829588.692775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829589.204754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829589.204849:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829589.396719:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829589.396721:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829589.716711:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829590.228760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829590.228835:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829590.420803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829590.420803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829590.740783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829591.252755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829591.252857:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829591.444792:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829591.444795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829591.444803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829591.444805:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829591.444814:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.444828:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2597 00000100:00080000:2.0:1787829591.444844:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.444855:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829591.444915:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518115584/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829591.444947:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.444962:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829591.444964:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.444969:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2597 00000100:00080000:2.0:1787829591.444973:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.444978:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829591.444990:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829591.445009:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518115712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829591.445026:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518115584:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829591.445029:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.445040:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829591.445046:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.445052:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2597 00000100:00080000:2.0:1787829591.445056:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.445061:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829591.445066:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829591.445103:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518115840/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829591.445116:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.445120:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829591.445122:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.445127:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2597 00000100:00080000:2.0:1787829591.445132:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.445137:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829591.445144:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829591.445162:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518115968/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829591.445171:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.445176:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829591.445178:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.445187:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2597 00000100:00100000:3.0:1787829591.445188:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518115712:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829591.445194:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.445206:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829591.445214:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829591.445233:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518116096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829591.445247:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.445251:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829591.445256:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829591.445258:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518115840:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787829591.445267:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2597 00000100:00080000:2.0:1787829591.445273:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.445279:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829591.445285:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829591.445298:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518116224/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829591.445308:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518115968:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787829591.445314:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829591.445321:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829591.445323:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829591.445329:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2597 00000100:00080000:2.0:1787829591.445335:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829591.445340:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829591.445346:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829591.445359:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518116352/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829591.445365:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518116096:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829591.445429:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518116224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829591.445456:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518116352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829591.764791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829592.276751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829592.276828:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829592.468765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829592.468833:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829592.788766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829593.300732:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829593.300792:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829593.492740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829593.492799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829593.812736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829594.324764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829594.324809:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829594.516796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829594.516913:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:2.0:1787829594.517023:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787829594.517070:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829594.517092:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829594.517142:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829594.517159:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829594.517171:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829594.517183:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829594.517193:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829594.517201:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829594] req@ffff8b2e8801ed00 x1874672518115712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787829594.517208:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829594.517237:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518115712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829594.517253:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829594.517259:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829594.517264:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518115712:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829594.517273:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829594] req@ffff8b2e8801c780 x1874672518115840/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829594.517289:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518115840/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829594.517298:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829594.517303:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829594.517307:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518115840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829594.517315:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829594] req@ffff8b2e8801f0c0 x1874672518115968/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829594.517328:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518115968/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829594.517337:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829594.517341:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829594.517345:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518115968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829594.517353:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829594] req@ffff8b2e8801d680 x1874672518115584/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829594.517367:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518115584/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829594.517376:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829594.517380:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829594.517384:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518115584:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829594.517420:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829594] req@ffff8b2e8801c3c0 x1874672518116352/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829594.517435:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518116352/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829594.517444:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829594.517448:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829594.517452:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518116352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829594.517459:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829594] req@ffff8b2e8801cb40 x1874672518116096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829594.517471:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518116096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829594.517481:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829594.517485:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829594.517489:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518116096:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829594.517496:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829591/real 1787829594] req@ffff8b2e8801de00 x1874672518116224/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829594.517509:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518116224/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829616 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829594.517518:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829594.517523:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829594.517526:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518116224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829594.836766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829595.348731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829595.348795:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829595.540772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829595.540817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829595.860766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829596.372754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829596.372804:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829596.564805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829596.564808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829596.564865:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829596.564869:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829596.564873:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829596.564880:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2602 00000100:00080000:3.0:1787829596.564888:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829596.564893:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829596.564935:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518116480/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829596.564959:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829596.564963:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829596.564965:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829596.564968:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2602 00000100:00080000:3.0:1787829596.564972:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829596.564976:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829596.564996:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829596.565008:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518116480:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829596.565011:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518116608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829596.565023:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829596.565026:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829596.565029:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829596.565034:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2602 00000100:00080000:3.0:1787829596.565038:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829596.565042:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829596.565054:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829596.565067:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518116736/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829596.565077:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829596.565080:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829596.565082:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829596.565085:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2602 00000100:00080000:3.0:1787829596.565088:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829596.565093:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829596.565094:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829596.565101:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829596.565113:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518116864/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829596.565121:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829596.565124:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829596.565126:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829596.565130:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2602 00000100:00080000:3.0:1787829596.565134:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829596.565139:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff8b2e8803b840 x1874672518116480/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829596.565139:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829596.565147:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829596.565160:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518116992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829596.565167:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518116480/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829596.565168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829596.565174:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829596.565176:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829596.565184:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2602 00000100:00080000:3.0:1787829596.565188:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829596.565191:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829596.565192:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829596.565200:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:3.0:1787829596.565201:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829596.565206:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518116480:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829596.565215:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518117120/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829596.565223:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829596.565226:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829596.565228:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829596.565234:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518116608:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829596.565234:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2602 00000100:00080000:3.0:1787829596.565240:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829596.565245:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829596.565250:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829596.565262:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518117248/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829596.565268:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829596.565288:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff8b2e8803b0c0 x1874672518116608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829596.565301:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518116608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829596.565315:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829596.565319:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829596.565323:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518116608:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829596.565329:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518116736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829596.565371:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829596.565383:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff8b2e8803a1c0 x1874672518116736/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829596.565395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518116736/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829596.565402:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829596.565405:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829596.565408:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518116736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829596.565414:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518116864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829596.565433:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829596.565443:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff8b2e88038f00 x1874672518116864/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829596.565453:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518116864/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829596.565460:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829596.565463:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829596.565466:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518116864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829596.565472:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518116992:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829596.565498:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829596.565509:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff8b2e8803a580 x1874672518116992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829596.565520:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518116992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829596.565528:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829596.565531:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829596.565534:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518116992:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829596.565540:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518117120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829596.565559:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829596.565570:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff8b2e880383c0 x1874672518117120/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829596.565580:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518117120/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829596.565587:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829596.565591:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829596.565594:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518117120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829596.565611:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518117248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829596.565629:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829596.565640:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829596/real 1787829596] req@ffff8b2e880392c0 x1874672518117248/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829596.565680:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518117248/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829621 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829596.565687:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829596.565690:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829596.565694:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518117248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829596.884731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829597.396782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829597.396813:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829597.588778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829597.588885:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829597.908783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829598.420754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829598.420780:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829598.612720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829598.612720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829598.932744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829599.444746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829599.444796:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829599.636740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829599.636740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829599.956754:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829600.468759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829600.468805:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829600.660783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829600.660815:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829600.980738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829601.492752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829601.492818:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829601.684769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829601.684769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829601.684784:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829601.684793:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829601.684800:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829601.684817:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2608 00000100:00080000:0.0:1787829601.684834:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829601.684844:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829601.684901:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518117376/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829601.684931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829601.684935:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829601.684937:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829601.684943:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2608 00000100:00080000:0.0:1787829601.684949:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829601.684954:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829601.684967:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829601.684971:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518117376:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829601.684987:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518117504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829601.685001:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829601.685006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829601.685008:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829601.685012:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2608 00000100:00080000:0.0:1787829601.685016:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829601.685020:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829601.685025:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829601.685037:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518117632/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829601.685046:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829601.685055:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829601.685059:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829601.685061:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829601.685067:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2608 00000100:00080000:0.0:1787829601.685072:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829601.685077:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829601.685083:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829601.685094:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff8b2e880421c0 x1874672518117376/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829601.685100:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518117760/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829601.685136:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829601.685141:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829601.685143:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829601.685151:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2608 00000100:00100000:3.0:1787829601.685151:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518117376/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829601.685160:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829601.685168:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829601.685169:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829601.685177:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:0.0:1787829601.685178:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829601.685181:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518117376:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829601.685197:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518117888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829601.685207:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518117504:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829601.685208:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829601.685215:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829601.685218:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829601.685225:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2608 00000100:00080000:0.0:1787829601.685230:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829601.685231:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829601.685240:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829601.685248:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829601.685248:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff8b2e88040b40 x1874672518117504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829601.685263:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518117504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829601.685269:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518118016/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829601.685272:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829601.685280:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829601.685286:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518117504:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829601.685294:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829601.685296:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518117632:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829601.685302:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829601.685307:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829601.685313:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2608 00000800:00000100:3.0:1787829601.685317:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829601.685319:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829601.685326:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829601.685329:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff8b2e880412c0 x1874672518117632/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829601.685333:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829601.685340:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518117632/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829601.685355:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829601.685358:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829601.685366:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518118144/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829601.685366:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518117632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829601.685381:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518117760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829601.685410:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829601.685424:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff8b2e88043840 x1874672518117760/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829601.685434:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518117760/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829601.685440:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829601.685443:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829601.685445:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518117760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829601.685456:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518117888:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829601.685471:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829601.685478:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff8b2e88040780 x1874672518117888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829601.685486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518117888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829601.685493:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829601.685495:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829601.685497:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518117888:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829601.685502:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518118016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829601.685512:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829601.685518:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff8b2e88042580 x1874672518118016/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829601.685525:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518118016/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829601.685530:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829601.685532:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829601.685534:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518118016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829601.685537:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518118144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829601.685546:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829601.685552:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829601/real 1787829601] req@ffff8b2e88042d00 x1874672518118144/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829601.685559:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518118144/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829626 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829601.685564:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829601.685566:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829601.685567:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518118144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829602.004737:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829602.516749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829602.516801:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829602.708777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829602.708777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829603.028733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829603.540781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829603.540837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829603.732769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829603.732772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829604.052721:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829604.564728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829604.564796:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829604.756752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829604.756752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829605.076789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829605.588748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829605.588791:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829605.780749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829605.780749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829606.100719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829606.612731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829606.612751:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829606.804730:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829606.804731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829606.804731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829606.804742:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829606.804747:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829606.804756:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2613 00000100:00080000:0.0:1787829606.804767:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829606.804776:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829606.804812:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518118272/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829606.804834:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829606.804837:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829606.804839:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829606.804842:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2613 00000100:00080000:0.0:1787829606.804861:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829606.804865:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829606.804873:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829606.804875:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518118272:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829606.804889:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518118400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829606.804899:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829606.804901:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829606.804903:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829606.804906:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2613 00000100:00080000:0.0:1787829606.804909:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829606.804912:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829606.804915:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829606.804924:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518118528/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829606.804931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829606.804933:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829606.804934:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829606.804939:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2613 00000800:00000100:3.0:1787829606.804940:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829606.804942:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829606.804946:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829606.804950:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829606.804958:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518118656/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829606.804964:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829606.804966:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829606.804968:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829606.804971:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2613 00000100:00080000:0.0:1787829606.804973:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829606.804976:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829606.804979:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829606.804986:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff8b2e88041e00 x1874672518118272/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829606.804989:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518118784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829606.805001:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829606.805008:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829606.805011:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829606.805016:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2613 00000100:00080000:0.0:1787829606.805019:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829606.805024:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829606.805025:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518118272/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829606.805027:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829606.805038:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518118912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829606.805039:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829606.805047:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829606.805052:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829606.805055:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518118272:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829606.805057:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829606.805060:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829606.805065:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2613 00000100:00080000:0.0:1787829606.805067:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829606.805070:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829606.805073:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829606.805080:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518119040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829606.805082:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518118400:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829606.805111:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829606.805122:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff8b2e88042d00 x1874672518118400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829606.805130:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518118400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829606.805143:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829606.805146:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829606.805148:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518118400:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829606.805153:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518118528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829606.805167:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829606.805174:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff8b2e88042580 x1874672518118528/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829606.805182:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518118528/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829606.805188:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829606.805190:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829606.805193:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518118528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829606.805196:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518118656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829606.805206:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829606.805211:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff8b2e88040780 x1874672518118656/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829606.805219:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518118656/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829606.805224:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829606.805227:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829606.805228:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518118656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829606.805232:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518118784:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829606.805241:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829606.805247:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff8b2e88043840 x1874672518118784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829606.805258:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518118784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829606.805264:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829606.805265:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829606.805267:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518118784:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829606.805271:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518118912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829606.805280:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829606.805286:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff8b2e880412c0 x1874672518118912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829606.805293:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518118912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829606.805298:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829606.805300:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829606.805302:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518118912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829606.805313:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518119040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829606.805323:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829606.805329:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829606/real 1787829606] req@ffff8b2e88040b40 x1874672518119040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829606.805336:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518119040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829631 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829606.805345:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829606.805347:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829606.805349:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518119040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829607.124751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829607.636782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829607.636810:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829607.828775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829607.828775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829608.148807:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829608.660778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829608.660817:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829608.852733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829608.852735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829609.172773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829609.684768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829609.684816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829609.876758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829609.876765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829610.196782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829610.708740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829610.708799:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829610.900756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829610.900764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829611.220733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829611.732756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829611.732808:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829611.924735:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829611.924740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829611.924743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829611.924747:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829611.924754:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829611.924765:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2618 00000100:00080000:0.0:1787829611.924779:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829611.924789:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829611.924853:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518119168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829611.924883:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829611.924886:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829611.924887:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829611.924891:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2618 00000100:00080000:0.0:1787829611.924895:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829611.924898:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829611.924909:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829611.924919:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518119168:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829611.924921:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518119296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829611.924933:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829611.924935:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829611.924936:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829611.924939:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2618 00000100:00080000:0.0:1787829611.924940:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829611.924943:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829611.924946:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829611.924953:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518119424/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829611.924960:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829611.924963:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829611.924964:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829611.924966:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2618 00000100:00080000:0.0:1787829611.924970:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829611.924973:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829611.924977:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829611.924985:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518119552/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829611.924991:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829611.924993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829611.924995:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829611.924998:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2618 00000100:00080000:0.0:1787829611.925000:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829611.925004:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829611.925007:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829611.925008:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829611.925018:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518119680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829611.925024:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829611.925026:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829611.925027:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829611.925030:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2618 00000100:00080000:0.0:1787829611.925038:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829611.925041:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829611.925044:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829611.925051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518119808/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829611.925062:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff8b2e880421c0 x1874672518119168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829611.925065:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829611.925074:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829611.925077:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829611.925082:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2618 00000100:00080000:0.0:1787829611.925085:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829611.925088:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829611.925091:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518119168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829611.925093:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829611.925105:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518119936/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829611.925118:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829611.925126:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829611.925132:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518119168:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829611.925160:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518119296:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829611.925185:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829611.925197:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff8b2e88040f00 x1874672518119296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829611.925206:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518119296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829611.925214:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829611.925216:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829611.925219:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518119296:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829611.925223:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518119424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829611.925236:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829611.925243:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff8b2e88041a40 x1874672518119424/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829611.925251:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518119424/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829611.925257:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829611.925260:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829611.925262:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518119424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829611.925265:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518119552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829611.925275:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829611.925281:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff8b2e88040000 x1874672518119552/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829611.925288:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518119552/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829611.925293:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829611.925295:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829611.925298:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518119552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829611.925302:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518119680:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829611.925317:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829611.925328:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff8b2e880430c0 x1874672518119680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829611.925341:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518119680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829611.925351:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829611.925354:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829611.925358:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518119680:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829611.925364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518119808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829611.925382:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829611.925393:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff8b2e88043480 x1874672518119808/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829611.925406:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518119808/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829611.925415:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829611.925419:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829611.925422:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518119808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829611.925429:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518119936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829611.925447:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829611.925456:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829611/real 1787829611] req@ffff8b2e88042940 x1874672518119936/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829611.925464:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518119936/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829636 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829611.925471:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829611.925473:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829611.925475:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518119936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829612.244822:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829612.756731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829612.756746:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829612.948779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829612.948779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829613.268730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829613.780780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829613.780829:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829613.972730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829613.972732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829614.292807:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829614.804754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829614.804817:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829614.996799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829614.996799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829615.316764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829615.828749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829615.828780:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829616.020713:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829616.020749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829616.340774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829616.852770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829616.852823:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787829617.044764:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829617.044767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829617.044770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829617.044788:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829617.044795:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829617.044805:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2623 00000100:00080000:2.0:1787829617.044816:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829617.044823:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829617.044866:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518120064/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829617.044893:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829617.044897:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829617.044899:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829617.044904:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2623 00000100:00080000:2.0:1787829617.044908:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829617.044913:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829617.044923:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829617.044933:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518120064:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829617.044939:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518120192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829617.044953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829617.044957:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829617.044959:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829617.044962:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2623 00000100:00080000:2.0:1787829617.044967:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829617.044971:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829617.044977:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829617.044990:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518120320/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829617.045002:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829617.045003:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829617.045009:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829617.045011:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829617.045017:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2623 00000100:00080000:2.0:1787829617.045022:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829617.045027:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829617.045032:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829617.045045:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518120448/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829617.045055:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829617.045059:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829617.045061:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829617.045066:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829617/real 1787829617] req@ffff8b2e8801de00 x1874672518120064/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829617.045069:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2623 00000100:00080000:2.0:1787829617.045077:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829617.045085:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829617.045091:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829617.045095:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518120064/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829617.045106:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518120576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829617.045141:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829617.045141:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829617.045147:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829617.045151:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829617.045155:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829617.045160:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2623 00000100:00100000:3.0:1787829617.045165:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518120064:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829617.045167:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829617.045176:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829617.045182:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829617.045195:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518120704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829617.045197:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518120192:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829617.045226:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829617.045230:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829617.045232:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829617.045236:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829617.045240:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2623 00000100:00080000:2.0:1787829617.045246:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829617.045251:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829617.045256:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829617.045262:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829617/real 1787829617] req@ffff8b2e8801cb40 x1874672518120192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829617.045271:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518120832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829617.045285:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518120192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829617.045301:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829617.045306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829617.045312:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518120192:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829617.045320:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518120320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829617.045342:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829617.045356:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829617/real 1787829617] req@ffff8b2e8801c3c0 x1874672518120320/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829617.045369:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518120320/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829617.045377:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829617.045380:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829617.045383:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518120320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829617.045390:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518120448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829617.045412:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829617.045424:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829617/real 1787829617] req@ffff8b2e8801d680 x1874672518120448/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829617.045436:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518120448/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829617.045445:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829617.045449:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829617.045453:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518120448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829617.045459:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518120576:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829617.045479:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829617.045501:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829617/real 1787829617] req@ffff8b2e8801f0c0 x1874672518120576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829617.045515:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518120576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829617.045524:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829617.045527:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829617.045530:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518120576:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829617.045549:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518120704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829617.045572:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829617.045585:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829617/real 1787829617] req@ffff8b2e8801c780 x1874672518120704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829617.045598:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518120704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829617.045606:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829617.045609:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829617.045613:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518120704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829617.045619:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518120832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829617.045640:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829617.045677:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829617/real 1787829617] req@ffff8b2e8801ed00 x1874672518120832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829617.045690:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518120832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829642 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829617.045699:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829617.045703:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829617.045706:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518120832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829617.364788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829617.876748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829617.876792:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829618.068785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829618.068785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829618.388850:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829618.900739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829618.900789:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829619.092733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829619.092735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829619.412710:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829619.924788:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829619.924889:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829620.116816:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829620.116816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829620.436690:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829620.948772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829620.948839:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829621.140771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829621.140802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829621.460766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829621.972785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829621.972836:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829622.164728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829622.164728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829622.164728:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829622.164749:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829622.164753:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829622.164762:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2628 00000100:00080000:2.0:1787829622.164772:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829622.164778:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829622.164819:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518120960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829622.164841:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829622.164843:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829622.164845:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829622.164848:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2628 00000100:00080000:2.0:1787829622.164851:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829622.164854:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829622.164862:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829622.164874:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518120960:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829622.164875:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518121088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829622.164889:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829622.164892:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829622.164893:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829622.164895:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2628 00000100:00080000:2.0:1787829622.164897:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829622.164900:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829622.164904:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829622.164914:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518121216/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829622.164926:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829622.164929:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829622.164930:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829622.164932:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2628 00000100:00080000:2.0:1787829622.164935:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829622.164939:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829622.164942:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829622.164947:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829622.164955:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518121344/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829622.164962:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829622.164965:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829622.164966:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829622.164969:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2628 00000100:00080000:2.0:1787829622.164971:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829622.164974:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829622.164977:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829622.164985:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829622/real 1787829622] req@ffff8b2e8801da40 x1874672518120960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829622.164987:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518121472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829622.165011:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829622.165013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829622.165016:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829622.165018:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518120960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829622.165022:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2628 00000100:00080000:2.0:1787829622.165029:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829622.165034:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829622.165036:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829622.165040:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:2.0:1787829622.165044:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829622.165045:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518120960:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829622.165064:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518121600/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829622.165071:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518121088:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829622.165084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829622.165087:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829622.165089:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829622.165094:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2628 00000800:00000100:3.0:1787829622.165096:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829622.165100:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829622.165106:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829622.165110:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829622/real 1787829622] req@ffff8b2e8801fc00 x1874672518121088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829622.165112:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829622.165125:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518121088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829622.165129:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518121728/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829622.165145:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829622.165151:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829622.165155:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518121088:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829622.165159:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518121216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829622.165174:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829622.165181:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829622/real 1787829622] req@ffff8b2e8801f480 x1874672518121216/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829622.165188:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518121216/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829622.165193:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829622.165196:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829622.165197:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518121216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829622.165201:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518121344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829622.165218:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829622.165226:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829622/real 1787829622] req@ffff8b2e8801cf00 x1874672518121344/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829622.165234:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518121344/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829622.165240:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829622.165242:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829622.165244:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518121344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829622.165247:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518121472:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829622.165258:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829622.165264:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829622/real 1787829622] req@ffff8b2e8801c000 x1874672518121472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829622.165271:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518121472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829622.165276:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829622.165278:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829622.165279:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518121472:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829622.165290:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518121600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829622.165300:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829622.165306:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829622/real 1787829622] req@ffff8b2e8801e580 x1874672518121600/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829622.165313:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518121600/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829622.165319:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829622.165321:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829622.165323:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518121600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829622.165326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518121728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829622.165335:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829622.165340:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829622/real 1787829622] req@ffff8b2e8801e940 x1874672518121728/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829622.165347:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518121728/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829647 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829622.165352:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829622.165354:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829622.165356:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518121728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829622.484733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829622.996779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829622.996839:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829623.188775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829623.188787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829623.508758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829624.020759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829624.020805:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829624.212759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829624.212759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829624.532711:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829625.044770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829625.044898:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829625.236745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829625.236745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829625.556726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829626.068791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829626.068836:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829626.260764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829626.260768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829626.580725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829627.092799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829627.092824:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829627.284783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829627.284785:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829627.284797:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829627.284802:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829627.284814:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2633 00000100:00080000:2.0:1787829627.284827:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829627.284828:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829627.284836:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829627.284878:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518121856/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829627.284902:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829627.284905:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829627.284907:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829627.284911:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2633 00000100:00080000:2.0:1787829627.284914:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829627.284917:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829627.284927:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829627.284934:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518121856:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829627.284940:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518121984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829627.284950:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829627.284954:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829627.284956:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829627.284959:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2633 00000100:00080000:2.0:1787829627.284961:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829627.284964:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829627.284967:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829627.284976:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518122112/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829627.284983:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829627.284986:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829627.284988:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829627.284991:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2633 00000100:00080000:2.0:1787829627.284994:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829627.284996:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829627.285000:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829627.285006:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829627.285013:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518122240/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829627.285021:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829627.285024:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829627.285025:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829627.285029:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2633 00000100:00080000:2.0:1787829627.285031:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829627.285034:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829627.285037:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829627.285045:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518122368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829627.285059:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829627.285062:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829627.285063:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829627.285070:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2633 00000100:00100000:3.0:1787829627.285071:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff8b2e8801e940 x1874672518121856/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829627.285073:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829627.285079:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829627.285086:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829627.285098:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518122496/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829627.285107:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829627.285109:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518121856/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829627.285111:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829627.285114:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829627.285121:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2633 00000100:00080000:2.0:1787829627.285127:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829627.285131:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829627.285132:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829627.285134:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829627.285140:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829627.285144:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518121856:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829627.285152:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518122624/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829627.285175:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518121984:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829627.285207:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829627.285222:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff8b2e8801e580 x1874672518121984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829627.285232:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518121984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.285239:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829627.285242:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829627.285244:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518121984:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829627.285249:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518122112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829627.285265:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829627.285272:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff8b2e8801c000 x1874672518122112/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829627.285290:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518122112/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.285300:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829627.285303:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829627.285305:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518122112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829627.285309:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518122240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829627.285320:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829627.285326:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff8b2e8801cf00 x1874672518122240/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829627.285334:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518122240/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.285340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829627.285342:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829627.285344:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518122240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829627.285348:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518122368:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829627.285359:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829627.285365:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff8b2e8801f480 x1874672518122368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829627.285373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518122368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.285379:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829627.285381:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829627.285384:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518122368:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829627.285388:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518122496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829627.285398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829627.285405:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff8b2e8801fc00 x1874672518122496/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829627.285412:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518122496/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.285432:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829627.285435:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829627.285437:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518122496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829627.285442:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518122624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829627.285454:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829627.285461:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829627/real 1787829627] req@ffff8b2e8801da40 x1874672518122624/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829627.285469:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518122624/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829652 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829627.285504:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829627.285507:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829627.285510:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518122624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829627.604791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829628.116798:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829628.116831:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829628.308780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829628.308780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829628.628761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829629.140787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829629.140835:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829629.332770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829629.332797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829629.652808:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829630.164790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829630.164828:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829630.356760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829630.356765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829630.676722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829631.188769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829631.188814:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829631.380752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829631.380763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829631.700769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829632.212775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829632.212843:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829632.404775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829632.404775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829632.404776:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829632.404790:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829632.404794:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829632.404806:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2638 00000100:00080000:2.0:1787829632.404817:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829632.404827:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829632.404865:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518122752/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829632.404902:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829632.404905:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829632.404906:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829632.404910:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2638 00000100:00080000:2.0:1787829632.404912:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829632.404915:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829632.404924:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829632.404935:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518122880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829632.404943:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829632.404945:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829632.404957:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829632.404959:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2638 00000100:00080000:2.0:1787829632.404961:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829632.404964:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829632.404966:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829632.404973:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518123008/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829632.404982:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518122752:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829632.404996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829632.404999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829632.405000:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829632.405003:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2638 00000100:00080000:2.0:1787829632.405006:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829632.405009:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829632.405011:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829632.405020:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518123136/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829632.405030:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829632.405044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829632.405045:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829632.405048:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2638 00000100:00080000:2.0:1787829632.405050:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829632.405052:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829632.405055:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829632.405061:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518123264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829632.405067:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829632.405069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829632.405070:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829632.405073:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2638 00000100:00080000:2.0:1787829632.405076:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829632.405079:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829632.405082:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829632.405091:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518123392/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829632.405098:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829632.405103:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787829632.405103:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829632.405112:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829632.405116:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2638 00000100:00080000:2.0:1787829632.405120:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829632.405123:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829632.405127:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829632.405134:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518123520/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829632.405174:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829632] req@ffff8b2e8801ed00 x1874672518122752/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829632.405197:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518122752/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829632.405226:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829632.405232:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829632.405236:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518122752:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829632.405245:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518122880:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829632.405273:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829632.405288:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829632] req@ffff8b2e8801c780 x1874672518122880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829632.405302:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518122880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829632.405313:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829632.405317:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829632.405321:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518122880:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829632.405350:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518123008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829632.405374:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829632.405399:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829632] req@ffff8b2e8801f0c0 x1874672518123008/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829632.405413:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518123008/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829632.405423:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829632.405427:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829632.405431:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518123008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829632.405438:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518123136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829632.405460:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829632.405472:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829632] req@ffff8b2e8801d680 x1874672518123136/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829632.405485:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518123136/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829632.405494:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829632.405498:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829632.405501:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518123136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829632.405509:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518123264:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829632.405530:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829632.405543:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829632] req@ffff8b2e8801c3c0 x1874672518123264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829632.405557:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518123264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829632.405567:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829632.405571:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829632.405575:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518123264:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829632.405582:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518123392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829632.405602:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829632.405613:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829632] req@ffff8b2e8801cb40 x1874672518123392/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829632.405625:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518123392/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829632.405635:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829632.405639:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829632.405643:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518123392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829632.405689:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518123520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829632.405714:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829632.405727:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829632/real 1787829632] req@ffff8b2e8801de00 x1874672518123520/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829632.405750:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518123520/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829657 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829632.405759:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829632.405763:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829632.405767:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518123520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829632.724726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829633.236799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829633.236890:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829633.428752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829633.428752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829633.748770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829634.260760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829634.260819:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829634.452759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829634.452759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829634.772772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829635.284715:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829635.284791:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829635.476747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829635.476792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829635.796786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829636.308740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829636.308795:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829636.500774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829636.500783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829636.820726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829637.332743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829637.332803:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829637.524747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829637.524757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829637.524762:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.524774:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829637.524778:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.524788:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2643 00000100:00080000:3.0:1787829637.524797:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.524803:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829637.524837:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518123648/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829637.524859:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.524862:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829637.524864:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.524867:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2643 00000100:00080000:3.0:1787829637.524870:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.524873:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829637.524881:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829637.524890:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518123648:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829637.524893:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518123776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829637.524903:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.524906:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829637.524907:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.524910:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2643 00000100:00080000:3.0:1787829637.524912:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.524914:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829637.524917:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829637.524925:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518123904/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829637.524932:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.524934:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829637.524936:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.524938:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2643 00000100:00080000:3.0:1787829637.524941:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.524943:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829637.524946:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829637.524949:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829637.524957:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518124032/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829637.524965:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.524967:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829637.524969:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.524972:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2643 00000100:00080000:3.0:1787829637.524974:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.524976:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829637.524985:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829637.524991:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff8b2e88039a40 x1874672518123648/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829637.524997:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518124160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829637.525009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.525011:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829637.525015:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518123648/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829637.525015:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.525019:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2643 00000100:00080000:3.0:1787829637.525025:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829637.525032:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829637.525035:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829637.525039:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:3.0:1787829637.525041:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829637.525044:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518123648:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829637.525052:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518124288/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829637.525059:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829637.525061:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829637.525063:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829637.525067:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2643 00000100:00100000:0.0:1787829637.525068:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518123776:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829637.525070:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829637.525081:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829637.525085:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829637.525094:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829637.525096:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518124416/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829637.525108:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff8b2e88039e00 x1874672518123776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829637.525118:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518123776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829637.525124:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829637.525127:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829637.525130:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518123776:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829637.525135:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518123904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829637.525148:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829637.525155:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff8b2e8803b480 x1874672518123904/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829637.525162:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518123904/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829637.525167:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829637.525169:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829637.525171:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518123904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829637.525174:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518124032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829637.525183:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829637.525189:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff8b2e88039680 x1874672518124032/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829637.525196:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518124032/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829637.525201:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829637.525203:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829637.525205:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518124032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829637.525208:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518124160:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829637.525217:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829637.525222:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff8b2e880392c0 x1874672518124160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829637.525233:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518124160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829637.525239:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829637.525240:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829637.525242:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518124160:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829637.525246:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518124288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829637.525255:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829637.525269:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff8b2e880383c0 x1874672518124288/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829637.525277:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518124288/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829637.525282:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829637.525284:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829637.525286:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518124288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829637.525298:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518124416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829637.525309:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829637.525314:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829637/real 1787829637] req@ffff8b2e8803a580 x1874672518124416/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829637.525322:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518124416/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829662 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829637.525331:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829637.525333:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829637.525335:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518124416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829637.844750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829638.356747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829638.356803:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829638.548750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829638.548750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829638.868773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829639.380746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829639.380782:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829639.572757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829639.572757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829639.892785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829640.404789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829640.404873:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829640.596775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829640.596784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829640.916751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829641.428741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829641.428838:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829641.620772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829641.620772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829641.940790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829642.452739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829642.452770:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829642.644740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829642.644750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829642.644862:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.644867:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829642.644871:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.644880:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2649 00000100:00080000:3.0:1787829642.644891:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.644899:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829642.644938:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518124544/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829642.644968:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.644971:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829642.644972:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.644975:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2649 00000100:00080000:3.0:1787829642.644978:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.644982:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829642.644995:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829642.645007:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518124544:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829642.645008:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518124672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829642.645029:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.645031:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829642.645033:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.645038:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2649 00000100:00080000:3.0:1787829642.645042:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.645045:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829642.645050:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829642.645074:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518124800/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829642.645087:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829642.645089:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829642.645096:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829642.645099:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.645106:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2649 00000100:00080000:3.0:1787829642.645112:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.645115:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829642.645121:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829642.645133:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff8b2e88038f00 x1874672518124544/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829642.645135:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518124928/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829642.645196:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.645200:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829642.645202:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829642.645208:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518124544/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829642.645213:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2649 00000100:00080000:3.0:1787829642.645220:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829642.645233:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829642.645235:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829642.645242:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829642.645247:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518124544:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787829642.645248:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829642.645271:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518125056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829642.645272:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518124672:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829642.645298:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829642.645301:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829642.645303:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829642.645311:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829642.645317:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2649 00000100:00080000:3.0:1787829642.645325:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.645332:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829642.645340:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff8b2e8803a1c0 x1874672518124672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829642.645343:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829642.645368:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518125184/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829642.645372:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518124672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829642.645412:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829642.645413:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829642.645424:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829642.645425:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787829642.645432:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518124672:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829642.645432:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829642.645450:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2649 00000100:00100000:0.0:1787829642.645455:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518124800:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787829642.645456:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829642.645468:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829642.645475:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829642.645486:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829642.645490:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518125312/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829642.645501:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff8b2e8803b0c0 x1874672518124800/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829642.645515:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518124800/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829642.645521:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829642.645524:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829642.645526:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518124800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829642.645530:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518124928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829642.645542:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829642.645548:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff8b2e8803b840 x1874672518124928/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829642.645555:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518124928/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829642.645560:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829642.645562:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829642.645564:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518124928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829642.645576:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518125056:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829642.645587:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829642.645611:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff8b2e8803a580 x1874672518125056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829642.645621:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518125056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829642.645627:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829642.645629:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829642.645631:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518125056:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829642.645636:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518125184:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829642.645678:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829642.645692:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff8b2e880383c0 x1874672518125184/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829642.645705:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518125184/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829642.645712:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829642.645714:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829642.645717:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518125184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829642.645721:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518125312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829642.645736:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829642.645744:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829642/real 1787829642] req@ffff8b2e880392c0 x1874672518125312/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829642.645756:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518125312/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829667 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829642.645761:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829642.645763:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829642.645765:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518125312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829642.964748:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829643.476800:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829643.476801:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829643.668718:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829643.668757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829643.988721:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787829644.500724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829644.500819:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829644.692720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829644.692720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829645.012726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829645.524778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829645.524800:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829645.716749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829645.716779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829646.036753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829646.548758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829646.548840:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829646.740783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829646.740784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829647.060703:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829647.572735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829647.572820:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829647.764764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829647.764776:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.764787:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787829647.764791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829647.764794:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.764816:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2654 00000100:00080000:3.0:1787829647.764832:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.764842:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829647.764912:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518125440/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829647.764953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.764975:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829647.764978:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.764984:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2654 00000100:00080000:3.0:1787829647.764988:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.764993:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829647.765005:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829647.765015:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518125440:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829647.765021:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518125568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829647.765032:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.765035:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829647.765037:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.765042:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2654 00000100:00080000:3.0:1787829647.765045:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.765048:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829647.765053:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829647.765063:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518125696/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829647.765070:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.765072:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829647.765074:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.765078:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2654 00000100:00080000:3.0:1787829647.765080:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.765083:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829647.765086:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829647.765095:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518125824/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829647.765099:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829647.765109:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.765113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829647.765115:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.765119:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2654 00000100:00080000:3.0:1787829647.765123:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.765126:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829647.765129:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829647.765138:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518125952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829647.765146:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.765149:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829647.765152:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff8b2e88039680 x1874672518125440/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829647.765153:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.765161:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2654 00000100:00080000:3.0:1787829647.765168:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.765176:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829647.765182:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518125440/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829647.765183:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829647.765198:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829647.765205:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829647.765207:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518126080/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829647.765210:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518125440:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829647.765227:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829647.765231:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829647.765233:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829647.765238:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2654 00000100:00100000:0.0:1787829647.765240:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518125568:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829647.765244:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829647.765250:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829647.765256:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829647.765267:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518126208/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829647.765276:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829647.765318:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff8b2e8803b480 x1874672518125568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829647.765345:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518125568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829647.765354:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829647.765357:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829647.765360:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518125568:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829647.765366:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518125696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829647.765388:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829647.765398:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff8b2e88039e00 x1874672518125696/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829647.765408:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518125696/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829647.765416:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829647.765419:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829647.765421:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518125696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829647.765426:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518125824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829647.765455:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829647.765464:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff8b2e88039a40 x1874672518125824/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829647.765475:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518125824/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829647.765490:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829647.765493:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829647.765496:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518125824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829647.765501:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518125952:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829647.765516:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829647.765524:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff8b2e880392c0 x1874672518125952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829647.765534:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518125952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829647.765543:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829647.765545:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829647.765548:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518125952:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829647.765553:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518126080:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829647.765593:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829647.765621:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff8b2e880383c0 x1874672518126080/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829647.765632:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518126080/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829647.765640:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829647.765643:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829647.765647:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518126080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829647.765689:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518126208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829647.765726:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829647.765742:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829647/real 1787829647] req@ffff8b2e8803a580 x1874672518126208/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829647.765765:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518126208/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829672 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829647.765796:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829647.765801:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829647.765807:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518126208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829648.084744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829648.596797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829648.596844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829648.788804:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829648.788804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829649.108759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829649.620760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829649.620823:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829649.812809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829649.812809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829650.132728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829650.644761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829650.644898:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829650.836747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829650.836765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829651.156791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829651.668752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829651.668820:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829651.860806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787829651.860807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829652.180779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829652.692756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829652.692844:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829652.884734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829652.884787:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829652.884788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829652.884799:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829652.884804:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.884814:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2659 00000100:00080000:1.0:1787829652.884825:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.884832:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829652.884876:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518126336/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829652.884909:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.884914:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829652.884916:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.884922:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2659 00000100:00080000:1.0:1787829652.884927:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.884931:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829652.884942:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829652.884954:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518126336:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829652.884960:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518126464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829652.884989:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.884993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829652.884996:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.885002:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2659 00000100:00080000:1.0:1787829652.885008:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.885013:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829652.885020:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829652.885033:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518126592/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829652.885040:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.885047:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787829652.885049:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829652.885050:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.885059:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2659 00000100:00080000:1.0:1787829652.885064:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.885068:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829652.885072:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829652.885081:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518126720/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829652.885088:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.885091:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829652.885093:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829652.885103:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff8b2e88074f00 x1874672518126336/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829652.885103:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2659 00000100:00080000:1.0:1787829652.885110:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.885121:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829652.885132:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829652.885140:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518126336/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829652.885144:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518126848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829652.885199:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829652.885199:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.885209:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829652.885215:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829652.885215:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.885237:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2659 00000100:00100000:0.0:1787829652.885238:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518126336:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829652.885251:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.885256:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829652.885261:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829652.885272:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518126976/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829652.885278:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518126464:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829652.885296:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829652.885299:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829652.885301:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829652.885307:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2659 00000800:00000100:0.0:1787829652.885318:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829652.885318:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829652.885327:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829652.885332:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829652.885337:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff8b2e88076940 x1874672518126464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829652.885345:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518127104/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829652.885366:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518126464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829652.885377:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829652.885380:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829652.885383:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518126464:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829652.885389:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518126592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829652.885405:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829652.885411:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff8b2e88077480 x1874672518126592/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829652.885419:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518126592/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829652.885424:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829652.885426:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829652.885428:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518126592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829652.885432:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518126720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829652.885453:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829652.885460:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff8b2e88075a40 x1874672518126720/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829652.885472:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518126720/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829652.885477:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829652.885479:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829652.885481:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518126720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829652.885484:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518126848:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829652.885494:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829652.885500:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff8b2e88075680 x1874672518126848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829652.885506:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518126848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829652.885512:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829652.885514:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829652.885515:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518126848:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829652.885543:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518126976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829652.885558:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829652.885570:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff8b2e88077c00 x1874672518126976/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829652.885578:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518126976/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829652.885592:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829652.885596:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829652.885600:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518126976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829652.885607:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518127104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829652.885626:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829652.885638:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829652/real 1787829652] req@ffff8b2e88074780 x1874672518127104/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829652.885668:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518127104/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829677 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829652.885674:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829652.885677:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829652.885679:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518127104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829653.204783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829653.716717:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829653.716776:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829653.908764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829653.908766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829654.228755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829654.740776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829654.740824:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829654.932710:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829654.932754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829655.252800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829655.764784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829655.764871:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829655.956790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829655.956802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829656.276768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829656.788764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829656.788820:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829656.980723:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829656.980724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829657.300723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829657.812761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829657.812788:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829658.004747:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829658.004753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829658.004754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829658.004760:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829658.004765:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829658.004774:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2664 00000100:00080000:0.0:1787829658.004784:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829658.004790:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829658.004826:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518127232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829658.004854:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829658.004857:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829658.004858:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829658.004861:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2664 00000100:00080000:0.0:1787829658.004863:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829658.004866:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829658.004874:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829658.004885:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518127360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829658.004893:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829658.004897:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829658.004899:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829658.004899:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518127232:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829658.004904:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2664 00000100:00080000:0.0:1787829658.004910:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829658.004912:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829658.004915:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829658.004922:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518127488/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829658.004928:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829658.004930:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829658.004932:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829658.004934:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2664 00000100:00080000:0.0:1787829658.004937:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829658.004939:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829658.004942:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829658.004949:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518127616/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829658.004955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829658.004957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829658.004959:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829658.004961:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2664 00000100:00080000:0.0:1787829658.004964:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829658.004967:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829658.004969:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829658.004976:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518127744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829658.004981:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829658.004984:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829658.004985:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829658.004988:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2664 00000100:00080000:0.0:1787829658.004990:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829658.004993:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829658.004996:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829658.005004:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518127872/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829658.005008:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518127360:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829658.005023:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829658.005026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829658.005027:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829658.005030:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2664 00000100:00080000:0.0:1787829658.005034:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829658.005037:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829658.005041:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829658.005046:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518127488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829658.005051:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518128000/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829658.005070:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518127616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829658.005084:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518127744:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829658.005099:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518127872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829658.005111:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518128000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829658.324759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829658.836839:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829658.836889:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829659.028788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829659.028829:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829659.348768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829659.860832:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829659.860925:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829660.052785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829660.052824:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829660.372725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829660.884775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829660.884801:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829661.076765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829661.076888:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:3.0:1787829661.077000:0:2336:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:3.0:1787829661.077053:0:2336:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829661.077067:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829661.077101:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829661.077108:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829661.077115:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829661.077121:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829661.077128:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829661.077137:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787829661.077158:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829658/real 1787829661] req@ffff8b2e88041a40 x1874672518127872/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829661.077184:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518127872/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829661.077196:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829661.077201:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829661.077206:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518127872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829661.077214:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829658/real 1787829661] req@ffff8b2e88040f00 x1874672518128000/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829661.077227:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518128000/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829661.077244:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829661.077247:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829661.077251:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518128000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829661.077264:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829658/real 1787829661] req@ffff8b2e88042940 x1874672518127360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829661.077277:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518127360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829661.077285:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829661.077289:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829661.077292:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518127360:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829661.077298:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829658/real 1787829661] req@ffff8b2e88043480 x1874672518127488/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829661.077311:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518127488/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829661.077323:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829661.077326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829661.077329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518127488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829661.077336:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829658/real 1787829661] req@ffff8b2e880430c0 x1874672518127616/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829661.077346:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518127616/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829661.077354:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829661.077357:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829661.077360:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518127616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829661.077367:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829658/real 1787829661] req@ffff8b2e88040000 x1874672518127744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829661.077377:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518127744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829661.077385:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829661.077388:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829661.077391:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518127744:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829661.077399:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829658/real 1787829661] req@ffff8b2e880403c0 x1874672518127232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829661.077411:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518127232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829683 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829661.077418:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829661.077420:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829661.077423:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518127232:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829661.396776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829661.908725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829661.908817:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829662.100776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829662.100837:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829662.420725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829662.932725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829662.932819:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829663.124754:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829663.124755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829663.124765:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829663.124769:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829663.124780:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2669 00000100:00080000:0.0:1787829663.124792:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829663.124801:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829663.124821:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787829663.124835:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518128128/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829663.124865:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829663.124868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829663.124870:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829663.124873:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2669 00000100:00080000:0.0:1787829663.124876:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829663.124878:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829663.124886:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829663.124896:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518128256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829663.124904:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829663.124907:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829663.124908:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829663.124911:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2669 00000100:00080000:0.0:1787829663.124913:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829663.124918:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829663.124918:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518128128:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829663.124923:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829663.124932:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518128384/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829663.124939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829663.124941:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829663.124943:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829663.124945:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2669 00000100:00080000:0.0:1787829663.124947:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829663.124949:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829663.124952:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829663.124961:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518128512/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829663.124967:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829663.124969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829663.124970:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829663.124973:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2669 00000100:00080000:0.0:1787829663.124977:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829663.124981:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829663.124986:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829663.124988:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829663.125000:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518128640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829663.125010:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829663.125014:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829663.125016:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829663.125020:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2669 00000100:00080000:0.0:1787829663.125024:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829663.125028:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829663.125033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829663.125043:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518128768/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829663.125052:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829663.125056:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829663/real 1787829663] req@ffff8b2e880403c0 x1874672518128128/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829663.125057:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829663.125061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829663.125070:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2669 00000100:00080000:0.0:1787829663.125077:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829663.125083:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829663.125087:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829663.125092:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518128128/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829663.125102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518128896/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829663.125109:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829663.125117:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829663.125123:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518128128:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829663.125133:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518128256:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829663.125159:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829663.125175:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829663/real 1787829663] req@ffff8b2e88040000 x1874672518128256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829663.125186:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518128256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829663.125194:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829663.125197:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829663.125201:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518128256:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829663.125226:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518128384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829663.125248:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829663.125261:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829663/real 1787829663] req@ffff8b2e880430c0 x1874672518128384/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829663.125274:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518128384/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829663.125283:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829663.125287:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829663.125290:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518128384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829663.125297:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518128512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829663.125316:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829663.125324:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829663/real 1787829663] req@ffff8b2e88043480 x1874672518128512/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829663.125333:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518128512/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829663.125342:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829663.125345:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829663.125349:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518128512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829663.125355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518128640:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829663.125376:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829663.125388:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829663/real 1787829663] req@ffff8b2e88042940 x1874672518128640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829663.125406:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518128640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829663.125415:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829663.125418:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829663.125422:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518128640:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829663.125428:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518128768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829663.125449:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829663.125478:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829663/real 1787829663] req@ffff8b2e88040f00 x1874672518128768/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829663.125491:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518128768/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829663.125500:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829663.125504:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829663.125508:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518128768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829663.125514:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518128896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829663.125536:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829663.125548:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829663/real 1787829663] req@ffff8b2e88041a40 x1874672518128896/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829663.125560:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518128896/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829688 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829663.125575:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829663.125578:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829663.125582:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518128896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829663.444736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829663.956766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829663.956819:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829664.148770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829664.148851:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829664.468709:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829664.980758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829664.980843:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829665.172761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829665.172775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829665.492705:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829666.004794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829666.004858:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829666.196753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829666.196758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829666.516699:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829667.028772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829667.028803:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829667.220716:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829667.220788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829667.540712:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829668.052759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829668.052856:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829668.244779:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829668.244791:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787829668.244793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829668.244794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829668.244796:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829668.244816:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2674 00000100:00080000:0.0:1787829668.244835:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829668.244843:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829668.244877:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518129024/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829668.244897:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829668.244900:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829668.244902:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829668.244906:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2674 00000100:00080000:0.0:1787829668.244909:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829668.244912:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829668.244922:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829668.244934:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518129152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829668.244942:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829668.244949:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829668.244949:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518129024:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829668.244956:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829668.244967:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2674 00000100:00080000:0.0:1787829668.244970:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829668.244972:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829668.244975:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829668.244983:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518129280/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829668.244989:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829668.244992:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829668.244994:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829668.244998:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2674 00000100:00080000:0.0:1787829668.245001:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829668.245003:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829668.245005:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829668.245014:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518129408/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829668.245021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829668.245024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829668.245025:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829668.245028:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2674 00000100:00080000:0.0:1787829668.245031:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829668.245033:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829668.245040:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829668.245044:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829668.245055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518129536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829668.245063:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829668.245066:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829668.245067:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829668.245071:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2674 00000100:00080000:0.0:1787829668.245074:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829668.245077:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829668.245080:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829668.245089:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518129664/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829668.245098:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff8b2e880421c0 x1874672518129024/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829668.245099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829668.245110:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829668.245112:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829668.245120:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2674 00000100:00080000:0.0:1787829668.245127:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829668.245131:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829668.245134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829668.245140:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518129792/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829668.245169:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518129024/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829668.245181:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829668.245186:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829668.245191:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518129024:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829668.245215:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518129152:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829668.245239:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829668.245250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff8b2e88040b40 x1874672518129152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829668.245260:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518129152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829668.245271:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829668.245275:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829668.245279:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518129152:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829668.245286:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518129280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829668.245312:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829668.245323:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff8b2e880412c0 x1874672518129280/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829668.245332:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518129280/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829668.245338:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829668.245341:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829668.245344:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518129280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829668.245348:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518129408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829668.245362:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829668.245368:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff8b2e88043840 x1874672518129408/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829668.245376:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518129408/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829668.245381:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829668.245383:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829668.245386:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518129408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829668.245390:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518129536:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829668.245402:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829668.245408:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff8b2e88040780 x1874672518129536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829668.245430:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518129536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829668.245437:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829668.245440:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829668.245442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518129536:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829668.245447:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518129664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829668.245462:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829668.245470:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff8b2e88042580 x1874672518129664/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829668.245478:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518129664/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829668.245484:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829668.245486:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829668.245488:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518129664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829668.245492:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518129792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829668.245502:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829668.245509:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829668/real 1787829668] req@ffff8b2e88042d00 x1874672518129792/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829668.245517:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518129792/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829693 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829668.245522:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829668.245524:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829668.245526:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518129792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829668.564713:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829669.076762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829669.076867:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829669.268702:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829669.268790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829669.588709:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829670.100738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829670.100774:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829670.292719:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829670.292743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829670.612710:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829671.124808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829671.124867:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829671.316769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829671.316769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829671.636764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829672.148768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829672.148852:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829672.340735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829672.340743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829672.660750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829673.172792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829673.172851:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829673.364768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829673.364834:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829673.364844:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829673.364847:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829673.364855:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829673.364870:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2679 00000100:00080000:3.0:1787829673.364885:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829673.364896:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829673.364952:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518129920/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829673.364978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829673.364981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829673.364983:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829673.364986:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2679 00000100:00080000:3.0:1787829673.364990:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829673.364995:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829673.365005:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829673.365018:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518129920:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829673.365019:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518130048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829673.365038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829673.365041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829673.365043:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829673.365046:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2679 00000100:00080000:3.0:1787829673.365048:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829673.365051:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829673.365055:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829673.365065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518130176/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829673.365072:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829673.365091:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829673.365093:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829673.365095:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2679 00000100:00080000:3.0:1787829673.365097:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829673.365100:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829673.365103:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829673.365112:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518130304/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829673.365119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829673.365120:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829673.365125:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829673.365128:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829673.365132:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2679 00000100:00080000:3.0:1787829673.365135:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829673.365139:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829673.365142:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829673.365151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518130432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829673.365158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829673.365185:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829673.365191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829673.365192:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff8b2e8803b840 x1874672518129920/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829673.365201:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2679 00000100:00080000:3.0:1787829673.365211:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829673.365218:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829673.365224:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829673.365236:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518130560/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829673.365244:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518129920/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829673.365274:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829673.365281:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829673.365283:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829673.365285:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829673.365288:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2679 00000100:00080000:0.0:1787829673.365293:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829673.365293:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829673.365301:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518129920:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829673.365302:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829673.365310:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829673.365318:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518130688/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829673.365358:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518130048:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829673.365442:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829673.365460:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff8b2e8803b0c0 x1874672518130048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829673.365473:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518130048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829673.365490:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829673.365494:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829673.365498:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518130048:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829673.365505:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518130176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829673.365526:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829673.365535:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff8b2e8803a1c0 x1874672518130176/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829673.365547:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518130176/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829673.365555:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829673.365558:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829673.365560:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518130176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829673.365566:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518130304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829673.365582:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829673.365592:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff8b2e88038f00 x1874672518130304/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829673.365603:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518130304/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829673.365611:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829673.365614:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829673.365618:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518130304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829673.365624:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518130432:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829673.365640:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829673.365648:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff8b2e8803a580 x1874672518130432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829673.365659:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518130432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829673.365699:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829673.365702:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829673.365705:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518130432:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829673.365712:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518130560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829673.365731:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829673.365742:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff8b2e880383c0 x1874672518130560/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829673.365753:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518130560/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829673.365762:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829673.365765:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829673.365768:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518130560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829673.365773:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518130688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829673.365788:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829673.365797:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829673/real 1787829673] req@ffff8b2e880392c0 x1874672518130688/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829673.365809:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518130688/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829698 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829673.365832:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829673.365833:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829673.365835:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518130688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829673.684776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829674.196760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829674.196848:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829674.388728:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829674.388751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829674.708753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829675.220719:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829675.220844:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829675.412768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829675.412768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829675.732768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829676.244749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829676.244766:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829676.436726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829676.436726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829676.756795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829677.268831:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829677.268886:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829677.460779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829677.460800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829677.780733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829678.292755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829678.292847:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829678.484784:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829678.484794:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829678.484799:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829678.484804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829678.484807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829678.484810:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2684 00000100:00080000:1.0:1787829678.484827:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829678.484836:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829678.484880:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518130816/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829678.484913:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829678.484917:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829678.484919:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829678.484923:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2684 00000100:00080000:1.0:1787829678.484925:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829678.484929:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829678.484939:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829678.484950:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518130944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829678.484956:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518130816:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829678.484960:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829678.484965:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829678.484967:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829678.484970:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2684 00000100:00080000:1.0:1787829678.484972:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829678.484975:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829678.484978:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829678.484988:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518131072/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829678.484995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829678.484997:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829678.484999:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829678.485001:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2684 00000100:00080000:1.0:1787829678.485019:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829678.485022:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829678.485024:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829678.485031:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829678.485035:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518131200/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829678.485043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829678.485046:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829678.485048:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829678.485051:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2684 00000100:00080000:1.0:1787829678.485054:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829678.485056:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829678.485059:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829678.485067:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518131328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829678.485074:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829678.485077:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff8b2e88077840 x1874672518130816/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829678.485078:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829678.485081:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829678.485094:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2684 00000100:00080000:1.0:1787829678.485097:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829678.485099:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518130816/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829678.485103:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829678.485111:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829678.485116:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829678.485122:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829678.485125:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518131456/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829678.485129:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518130816:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829678.485142:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829678.485145:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829678.485146:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829678.485151:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2684 00000100:00080000:1.0:1787829678.485153:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829678.485155:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829678.485158:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829678.485165:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518130944:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829678.485168:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518131584/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829678.485201:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829678.485216:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff8b2e88076580 x1874672518130944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829678.485229:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518130944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829678.485240:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829678.485243:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829678.485247:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518130944:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829678.485253:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518131072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829678.485275:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829678.485287:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff8b2e88076d00 x1874672518131072/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829678.485298:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518131072/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829678.485306:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829678.485309:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829678.485312:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518131072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829678.485318:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518131200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829678.485335:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829678.485344:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff8b2e880770c0 x1874672518131200/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829678.485371:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518131200/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829678.485380:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829678.485383:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829678.485388:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518131200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829678.485394:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518131328:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829678.485415:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829678.485428:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff8b2e88074000 x1874672518131328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829678.485438:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518131328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829678.485445:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829678.485447:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829678.485449:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518131328:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829678.485454:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518131456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829678.485468:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829678.485475:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff8b2e880761c0 x1874672518131456/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829678.485483:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518131456/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829678.485488:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829678.485490:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829678.485492:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518131456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829678.485505:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518131584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829678.485517:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829678.485524:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829678/real 1787829678] req@ffff8b2e88075e00 x1874672518131584/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829678.485531:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518131584/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829703 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829678.485537:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829678.485539:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829678.485540:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518131584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829678.804754:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829679.316759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829679.316807:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829679.508787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829679.508802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829679.828729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787829680.340759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829680.340842:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829680.532761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829680.532761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829680.852774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829681.364929:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829681.364933:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829681.556778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829681.556778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829681.876714:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829682.388753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829682.388812:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829682.580727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829682.580727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829682.900756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829683.412747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829683.412829:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829683.604733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829683.604733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829683.604763:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829683.604770:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829683.604775:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829683.604785:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2690 00000100:00080000:0.0:1787829683.604796:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.604801:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829683.604836:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518131712/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829683.604860:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829683.604863:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829683.604864:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829683.604867:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2690 00000100:00080000:0.0:1787829683.604869:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.604872:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829683.604881:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829683.604894:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518131840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829683.604897:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518131712:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829683.604915:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829683.604920:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829683.604922:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829683.604928:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2690 00000100:00080000:0.0:1787829683.604933:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.604937:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829683.604944:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829683.604959:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518131968/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829683.604959:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829683.604970:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829683.604972:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829683.604974:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829683.604978:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2690 00000100:00080000:0.0:1787829683.604981:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.604990:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829683.604995:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829683.605004:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518132096/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829683.605008:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff8b2e88041e00 x1874672518131712/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829683.605015:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829683.605022:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829683.605026:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829683.605033:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2690 00000100:00080000:0.0:1787829683.605037:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.605040:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829683.605043:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829683.605047:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518131712/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829683.605054:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518132224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829683.605060:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829683.605064:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829683.605068:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829683.605069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829683.605073:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829683.605076:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518131712:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829683.605080:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2690 00000100:00080000:0.0:1787829683.605087:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.605090:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829683.605092:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829683.605099:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518132352/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829683.605106:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518131840:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829683.605108:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829683.605115:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829683.605119:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829683.605123:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2690 00000100:00080000:0.0:1787829683.605125:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829683.605128:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829683.605131:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829683.605142:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829683.605143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518132480/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829683.605160:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff8b2e88042d00 x1874672518131840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829683.605188:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518131840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829683.605195:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829683.605198:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829683.605200:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518131840:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829683.605206:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518131968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829683.605222:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829683.605246:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff8b2e88042580 x1874672518131968/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829683.605257:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518131968/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829683.605264:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829683.605267:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829683.605269:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518131968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829683.605275:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518132096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829683.605291:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829683.605299:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff8b2e88040780 x1874672518132096/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829683.605342:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518132096/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829683.605350:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829683.605353:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829683.605356:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518132096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829683.605378:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518132224:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829683.605398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829683.605410:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff8b2e88043840 x1874672518132224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829683.605424:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518132224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829683.605433:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829683.605435:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829683.605437:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518132224:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829683.605451:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518132352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829683.605466:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829683.605473:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff8b2e880412c0 x1874672518132352/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829683.605481:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518132352/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829683.605490:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829683.605493:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829683.605497:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518132352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829683.605502:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518132480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829683.605521:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829683.605533:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829683/real 1787829683] req@ffff8b2e88040b40 x1874672518132480/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829683.605546:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518132480/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829708 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829683.605552:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829683.605555:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829683.605557:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518132480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829683.924706:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829684.436721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829684.436831:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829684.628711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829684.628719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829684.948745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829685.460716:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829685.460763:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829685.652789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829685.652789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829685.972717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829686.484752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829686.484826:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829686.676730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829686.676780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829686.996729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829687.508772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829687.508873:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829687.700769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829687.700770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829688.020778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829688.532758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829688.532858:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829688.724763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829688.724775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829688.724783:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829688.724789:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829688.724794:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829688.724804:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2695 00000100:00080000:3.0:1787829688.724814:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829688.724821:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829688.724858:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518132608/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829688.724887:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829688.724891:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829688.724893:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829688.724898:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2695 00000100:00080000:3.0:1787829688.724902:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829688.724906:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829688.724916:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829688.724920:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518132608:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829688.724927:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518132736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829688.724937:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829688.724940:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829688.724941:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829688.724944:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2695 00000100:00080000:3.0:1787829688.724947:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829688.724951:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829688.724955:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829688.724963:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518132864/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829688.724970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829688.724973:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829688.724975:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829688.724976:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829688.724978:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2695 00000100:00080000:3.0:1787829688.724981:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829688.724984:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829688.724988:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829688.725008:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518132992/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829688.725017:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829688.725019:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff8b2e88039a40 x1874672518132608/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829688.725020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829688.725022:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829688.725026:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2695 00000100:00080000:3.0:1787829688.725029:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829688.725033:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829688.725035:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518132608/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829688.725037:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829688.725045:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829688.725046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518133120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829688.725049:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829688.725054:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518132608:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829688.725055:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829688.725058:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829688.725060:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829688.725063:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2695 00000100:00080000:3.0:1787829688.725066:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829688.725069:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829688.725073:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829688.725075:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518132736:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829688.725082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518133248/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829688.725096:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829688.725107:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff8b2e88039e00 x1874672518132736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829688.725119:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518132736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829688.725128:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829688.725131:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829688.725135:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518132736:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829688.725142:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518132864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829688.725163:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829688.725169:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829688.725172:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829688.725174:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829688.725178:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2695 00000100:00080000:3.0:1787829688.725181:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829688.725184:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829688.725188:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff8b2e8803b480 x1874672518132864/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829688.725189:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829688.725201:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518132864/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829688.725201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518133376/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829688.725211:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829688.725214:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829688.725217:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518132864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829688.725224:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518132992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829688.725243:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829688.725255:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff8b2e88039680 x1874672518132992/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829688.725264:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518132992/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829688.725269:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829688.725271:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829688.725284:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518132992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829688.725289:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518133120:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829688.725303:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829688.725311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff8b2e880392c0 x1874672518133120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829688.725319:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518133120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829688.725324:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829688.725327:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829688.725329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518133120:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829688.725340:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518133248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829688.725351:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829688.725357:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff8b2e880383c0 x1874672518133248/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829688.725364:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518133248/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829688.725369:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829688.725371:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829688.725373:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518133248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829688.725376:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518133376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829688.725385:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829688.725390:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829688/real 1787829688] req@ffff8b2e8803a580 x1874672518133376/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829688.725397:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518133376/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829713 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829688.725406:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829688.725408:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829688.725410:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518133376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829689.044721:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829689.556777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829689.556852:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829689.748746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829689.748761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829690.068791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829690.580750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829690.580830:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829690.772762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829690.772762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829691.092720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829691.604739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829691.604838:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829691.796724:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829691.796736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829692.116837:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829692.628794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829692.628881:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829692.820796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829692.820857:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829693.140766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829693.652772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829693.652915:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829693.844771:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829693.844781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829693.844781:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829693.844788:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829693.844796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829693.844799:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2700 00000100:00080000:3.0:1787829693.844810:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829693.844819:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829693.844854:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518133504/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829693.844875:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829693.844878:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829693.844879:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829693.844882:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2700 00000100:00080000:3.0:1787829693.844885:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829693.844889:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829693.844897:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829693.844925:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518133632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829693.844932:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518133504:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829693.844952:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829693.844956:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829693.844957:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829693.844961:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2700 00000100:00080000:3.0:1787829693.844965:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829693.844968:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829693.844973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829693.844983:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518133760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829693.844990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829693.845009:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829693.845011:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829693.845014:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2700 00000100:00080000:3.0:1787829693.845017:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829693.845019:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829693.845024:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829693.845034:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518133888/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829693.845040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829693.845049:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829693.845050:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829693.845075:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829693.845081:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2700 00000100:00080000:3.0:1787829693.845086:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829693.845093:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829693.845100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829693.845120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518134016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829693.845142:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff8b2e88038f00 x1874672518133504/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829693.845180:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829693.845184:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829693.845187:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829693.845191:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518133504/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829693.845192:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2700 00000100:00080000:3.0:1787829693.845204:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829693.845215:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829693.845219:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829693.845222:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829693.845230:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829693.845249:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518133504:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829693.845256:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518134144/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829693.845266:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829693.845270:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829693.845272:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829693.845277:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2700 00000100:00080000:3.0:1787829693.845281:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829693.845287:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518133632:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829693.845289:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829693.845301:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829693.845323:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829693.845324:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518134272/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829693.845345:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff8b2e8803a1c0 x1874672518133632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829693.845358:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518133632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829693.845367:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829693.845370:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829693.845373:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518133632:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829693.845378:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518133760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829693.845393:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829693.845400:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff8b2e8803b0c0 x1874672518133760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829693.845408:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518133760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829693.845414:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829693.845416:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829693.845418:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518133760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829693.845422:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518133888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829693.845432:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829693.845438:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff8b2e8803b840 x1874672518133888/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829693.845445:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518133888/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829693.845450:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829693.845452:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829693.845455:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518133888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829693.845459:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518134016:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829693.845471:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829693.845476:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff8b2e8803a580 x1874672518134016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829693.845484:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518134016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829693.845489:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829693.845491:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829693.845493:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518134016:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829693.845497:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518134144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829693.845506:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829693.845512:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff8b2e880383c0 x1874672518134144/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829693.845519:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518134144/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829693.845524:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829693.845526:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829693.845527:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518134144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829693.845540:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518134272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829693.845559:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829693.845567:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829693/real 1787829693] req@ffff8b2e880392c0 x1874672518134272/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829693.845575:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518134272/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829718 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829693.845580:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829693.845583:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829693.845585:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518134272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829694.164761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829694.676790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829694.676858:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829694.868824:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829694.868825:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829695.188747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829695.700740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829695.700810:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829695.892716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829695.892745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829696.212790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829696.724767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829696.724875:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829696.916803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829696.916804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829697.236731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829697.748733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829697.748843:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829697.940752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829697.940773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829698.260776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829698.772786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829698.772881:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829698.964780:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.964791:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829698.964795:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.964809:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2705 00000100:00100000:0.0:1787829698.964812:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829698.964813:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829698.964820:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.964830:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829698.964868:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518134400/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.964892:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.964894:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829698.964896:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.964899:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2705 00000100:00080000:3.0:1787829698.964901:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.964904:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829698.964913:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829698.964922:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518134528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.964930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.964932:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829698.964934:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829698.964941:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518134400:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829698.964942:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2705 00000100:00080000:3.0:1787829698.964949:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.964958:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829698.964962:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829698.964971:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518134656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.964977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.964980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829698.964981:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.964984:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2705 00000100:00080000:3.0:1787829698.964986:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.964988:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829698.964991:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829698.965001:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518134784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.965007:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.965009:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829698.965010:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.965014:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2705 00000100:00080000:3.0:1787829698.965018:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.965020:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829698.965022:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829698.965029:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518134912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.965039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.965041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829698.965043:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829698.965048:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829698.965048:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2705 00000100:00080000:3.0:1787829698.965065:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.965069:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829698.965073:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829698.965083:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518135040/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829698.965091:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829698.965095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829698.965096:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829698.965102:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2705 00000100:00080000:3.0:1787829698.965107:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829698.965115:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829698.965117:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829698] req@ffff8b2e88039680 x1874672518134400/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829698.965128:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829698.965144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518135168/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829698.965153:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518134400/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829698.965175:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829698.965180:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829698.965184:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518134400:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829698.965233:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518134528:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829698.965255:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829698.965267:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829698] req@ffff8b2e8803b480 x1874672518134528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829698.965276:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518134528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829698.965283:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829698.965286:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829698.965288:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518134528:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829698.965293:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518134656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829698.965315:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829698.965323:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829698] req@ffff8b2e88039e00 x1874672518134656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829698.965331:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518134656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829698.965337:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829698.965339:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829698.965342:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518134656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829698.965346:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518134784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829698.965362:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829698.965369:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829698] req@ffff8b2e88039a40 x1874672518134784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829698.965377:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518134784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829698.965383:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829698.965386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829698.965388:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518134784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829698.965392:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518134912:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829698.965410:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829698.965417:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829698] req@ffff8b2e880392c0 x1874672518134912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829698.965424:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518134912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829698.965430:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829698.965433:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829698.965435:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518134912:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829698.965439:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518135040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829698.965449:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829698.965455:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829698] req@ffff8b2e880383c0 x1874672518135040/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829698.965462:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518135040/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829698.965468:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829698.965471:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829698.965476:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518135040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829698.965483:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518135168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829698.965513:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829698.965526:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829698/real 1787829698] req@ffff8b2e8803a580 x1874672518135168/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829698.965537:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518135168/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829723 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829698.965543:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829698.965545:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829698.965548:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518135168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829699.284763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829699.796737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829699.796794:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829699.988773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829699.988774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829700.308745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829700.820739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829700.820788:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829701.012820:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829701.012820:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829701.332731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829701.844742:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829701.844824:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829702.036760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829702.036761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829702.356742:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829702.868739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829702.868856:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829703.060767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829703.060768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829703.380726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829703.892739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829703.892804:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829704.084760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829704.084762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829704.084774:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829704.084781:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829704.084786:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829704.084797:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2710 00000100:00080000:2.0:1787829704.084810:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829704.084817:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829704.084862:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518135296/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829704.084884:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829704.084887:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829704.084888:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829704.084892:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2710 00000100:00080000:2.0:1787829704.084895:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829704.084898:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829704.084919:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829704.084931:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518135296:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829704.084931:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518135424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829704.084946:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829704.084948:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829704.084950:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829704.084953:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2710 00000100:00080000:2.0:1787829704.084955:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829704.084958:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829704.084961:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829704.084970:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518135552/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829704.084976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829704.084979:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829704.084980:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829704.084983:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2710 00000100:00080000:2.0:1787829704.084985:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829704.084987:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829704.084990:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829704.084997:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518135680/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829704.085002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829704.085004:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829704.085005:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829704.085008:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2710 00000100:00080000:2.0:1787829704.085011:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829704.085013:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829704.085017:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829704.085021:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829704.085029:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518135808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829704.085038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829704.085040:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829704.085042:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829704.085045:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2710 00000100:00080000:2.0:1787829704.085048:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829704.085050:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829704.085053:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829704.085060:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518135936/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829704.085066:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829704.085069:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829704.085070:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829704.085073:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829704/real 1787829704] req@ffff8b2e8801de00 x1874672518135296/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829704.085076:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2710 00000100:00080000:2.0:1787829704.085082:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829704.085089:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829704.085093:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829704.085098:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518135296/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829704.085103:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518136064/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829704.085116:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829704.085121:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829704.085128:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518135296:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829704.085155:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518135424:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829704.085196:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829704.085212:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829704/real 1787829704] req@ffff8b2e8801cb40 x1874672518135424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829704.085236:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518135424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829704.085243:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829704.085245:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829704.085248:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518135424:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829704.085253:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518135552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829704.085271:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829704.085278:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829704/real 1787829704] req@ffff8b2e8801c3c0 x1874672518135552/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829704.085286:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518135552/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829704.085301:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829704.085303:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829704.085305:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518135552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829704.085309:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518135680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829704.085320:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829704.085326:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829704/real 1787829704] req@ffff8b2e8801d680 x1874672518135680/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829704.085334:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518135680/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829704.085340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829704.085341:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829704.085343:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518135680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829704.085347:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518135808:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829704.085357:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829704.085363:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829704/real 1787829704] req@ffff8b2e8801f0c0 x1874672518135808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829704.085371:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518135808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829704.085376:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829704.085378:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829704.085379:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518135808:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829704.085383:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518135936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829704.085403:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829704.085409:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829704/real 1787829704] req@ffff8b2e8801c780 x1874672518135936/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829704.085416:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518135936/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829704.085422:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829704.085424:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829704.085427:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518135936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829704.085430:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518136064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829704.085442:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829704.085452:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829704/real 1787829704] req@ffff8b2e8801ed00 x1874672518136064/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829704.085462:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518136064/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829729 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829704.085471:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829704.085476:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829704.085479:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518136064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829704.404749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829704.916751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829704.916800:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829705.108770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829705.108770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829705.428757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829705.940728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829705.940794:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829706.132746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829706.132746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829706.452767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829706.964748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829706.964832:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829707.156737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829707.156787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829707.476776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829707.988741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829707.988807:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829708.180716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829708.180750:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829708.500720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829709.012775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829709.012846:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829709.204798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829709.204842:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829709.204851:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787829709.204851:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829709.204857:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829709.204875:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2715 00000100:00080000:1.0:1787829709.204888:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829709.204896:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829709.204939:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518136192/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829709.204969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829709.204974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829709.204976:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829709.204982:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2715 00000100:00080000:1.0:1787829709.204987:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829709.204992:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829709.204998:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518136192:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787829709.205004:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829709.205025:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518136320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829709.205036:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829709.205040:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829709.205042:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829709.205046:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2715 00000100:00080000:1.0:1787829709.205050:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829709.205055:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829709.205059:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829709.205067:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829709.205081:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518136448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829709.205092:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829709.205097:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829709/real 1787829709] req@ffff8b2e88074b40 x1874672518136192/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829709.205098:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829709.205102:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829709.205112:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2715 00000100:00080000:1.0:1787829709.205118:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829709.205121:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518136192/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829709.205126:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829709.205149:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829709.205152:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829709.205157:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829709.205161:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518136192:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829709.205167:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518136576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829709.205178:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829709.205184:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518136320:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829709.205184:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829709.205190:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829709.205197:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2715 00000100:00080000:1.0:1787829709.205201:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829709.205207:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787829709.205212:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787829709.205216:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829709.205229:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829709/real 1787829709] req@ffff8b2e88075e00 x1874672518136320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829709.205234:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518136704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829709.205259:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518136320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829709.205259:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829709.205267:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829709.205272:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829709.205279:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829709.205280:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2715 00000100:00080000:0.0:1787829709.205287:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829709.205288:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829709.205292:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518136320:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829709.205297:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829709.205302:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518136448:192.168.203.101@tcp:8:kworker.0 00000008:00100000:1.0:1787829709.205307:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829709.205325:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829709.205337:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829709/real 1787829709] req@ffff8b2e880761c0 x1874672518136448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829709.205344:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518136448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829709.205350:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829709.205353:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829709.205354:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518136448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829709.205359:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518136576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829709.205364:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518136832/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829709.205373:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829709.205382:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829709/real 1787829709] req@ffff8b2e88074000 x1874672518136576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829709.205383:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829709.205393:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829709.205397:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829709.205401:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518136576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829709.205406:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2715 00000100:00080000:0.0:1787829709.205411:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829709.205415:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829709.205416:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829709.205421:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518136576:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829709.205426:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829709.205433:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829709.205438:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518136704:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829709.205450:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518136960/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829709.205455:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829709.205465:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829709/real 1787829709] req@ffff8b2e880770c0 x1874672518136704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829709.205477:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518136704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829709.205483:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829709.205486:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829709.205488:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518136704:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829709.205493:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518136832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829709.205502:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829709.205508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829709/real 1787829709] req@ffff8b2e88076d00 x1874672518136832/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829709.205516:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518136832/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829709.205521:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829709.205522:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829709.205524:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518136832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829709.205535:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518136960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829709.205553:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829709.205565:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829709/real 1787829709] req@ffff8b2e88076580 x1874672518136960/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829709.205578:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518136960/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829734 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829709.205586:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829709.205588:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829709.205590:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518136960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829709.524743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829710.036764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829710.036800:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829710.228730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829710.228762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829710.548717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829711.060763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829711.060827:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829711.252733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829711.252736:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829711.572769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829712.084755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829712.084821:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829712.276771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829712.276771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829712.596717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829713.108739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829713.108837:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829713.300730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829713.300752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829713.620763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829714.132748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829714.132857:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829714.324752:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829714.324760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829714.324760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829714.324762:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829714.324769:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829714.324780:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2720 00000100:00080000:1.0:1787829714.324792:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829714.324799:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829714.324848:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518137088/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829714.324869:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829714.324872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829714.324874:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829714.324877:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2720 00000100:00080000:1.0:1787829714.324880:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829714.324883:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829714.324893:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829714.324904:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518137088:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829714.324910:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518137216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829714.324919:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829714.324922:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829714.324924:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829714.324928:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2720 00000100:00080000:1.0:1787829714.324932:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829714.324937:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829714.324942:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829714.324956:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518137344/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829714.324967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829714.324971:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829714.324973:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829714.324978:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2720 00000100:00080000:1.0:1787829714.324982:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829714.324983:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829714.324991:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829714.324997:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829714.325010:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518137472/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829714.325017:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829714.325020:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829714.325021:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829714.325025:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2720 00000100:00080000:1.0:1787829714.325028:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829714.325034:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829714.325035:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff8b2e88077840 x1874672518137088/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787829714.325037:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829714.325053:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518137600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829714.325059:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518137088/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829714.325064:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829714.325072:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829714.325076:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829714.325080:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829714.325082:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2720 00000100:00080000:1.0:1787829714.325085:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829714.325086:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829714.325090:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829714.325093:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518137088:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787829714.325117:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829714.325128:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518137728/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829714.325137:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518137216:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829714.325137:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829714.325145:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829714.325147:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829714.325151:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2720 00000100:00080000:1.0:1787829714.325154:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829714.325157:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829714.325161:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829714.325168:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829714.325173:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518137856/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829714.325182:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff8b2e88074780 x1874672518137216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829714.325193:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518137216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829714.325201:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829714.325203:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829714.325206:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518137216:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829714.325210:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518137344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829714.325224:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829714.325231:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff8b2e88077c00 x1874672518137344/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829714.325239:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518137344/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829714.325249:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829714.325251:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829714.325253:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518137344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829714.325257:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518137472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829714.325266:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829714.325272:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff8b2e88075680 x1874672518137472/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829714.325279:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518137472/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829714.325285:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829714.325287:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829714.325289:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518137472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829714.325293:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518137600:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829714.325302:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829714.325307:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff8b2e88075a40 x1874672518137600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829714.325315:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518137600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829714.325320:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829714.325322:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829714.325325:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518137600:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829714.325336:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518137728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829714.325347:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829714.325353:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff8b2e88077480 x1874672518137728/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829714.325361:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518137728/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829714.325366:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829714.325367:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829714.325369:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518137728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829714.325373:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518137856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829714.325382:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829714.325387:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829714/real 1787829714] req@ffff8b2e88076940 x1874672518137856/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829714.325395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518137856/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829739 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829714.325400:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829714.325402:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829714.325403:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518137856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829714.644744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829715.156752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829715.156843:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829715.348795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829715.348796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829715.668773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829716.180760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829716.180822:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829716.372781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829716.372782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829716.692728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829717.204734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829717.204859:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829717.396769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829717.396798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829717.716767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829718.228771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829718.228854:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829718.420753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829718.420753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829718.740706:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829719.252738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829719.252846:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829719.444763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829719.444773:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829719.444785:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829719.444790:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829719.444800:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2725 00000100:00100000:3.0:1787829719.444807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829719.444813:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829719.444823:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829719.444865:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518137984/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829719.444893:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829719.444896:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829719.444897:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829719.444901:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2725 00000100:00080000:1.0:1787829719.444904:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829719.444906:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829719.444916:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829719.444927:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518138112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829719.444934:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518137984:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829719.444939:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829719.444944:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829719.444945:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829719.444949:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2725 00000100:00080000:1.0:1787829719.444951:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829719.444953:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829719.444957:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829719.444966:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518138240/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829719.444972:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829719.444975:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829719.444976:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829719.444979:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2725 00000100:00080000:1.0:1787829719.444981:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829719.444983:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829719.444986:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829719.444993:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518138368/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829719.444998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829719.445000:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829719.445002:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829719.445005:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2725 00000100:00080000:1.0:1787829719.445008:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829719.445011:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829719.445014:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829719.445021:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829719.445023:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518138496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829719.445033:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829719.445035:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829719.445037:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829719.445041:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2725 00000100:00080000:1.0:1787829719.445050:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829719.445053:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829719.445056:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829719.445077:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518138624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829719.445084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829719.445087:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829719.445088:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829719.445093:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2725 00000100:00100000:0.0:1787829719.445095:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff8b2e88074f00 x1874672518137984/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829719.445096:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829719.445103:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829719.445108:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829719.445119:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518138752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829719.445128:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518137984/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829719.445143:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829719.445149:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829719.445154:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518137984:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829719.445183:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518138112:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829719.445212:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829719.445239:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff8b2e88076940 x1874672518138112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829719.445254:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518138112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829719.445264:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829719.445268:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829719.445272:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518138112:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829719.445279:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518138240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829719.445307:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829719.445320:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff8b2e88077480 x1874672518138240/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829719.445344:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518138240/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829719.445353:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829719.445357:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829719.445361:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518138240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829719.445368:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518138368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829719.445390:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829719.445402:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff8b2e88075a40 x1874672518138368/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829719.445415:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518138368/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829719.445424:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829719.445428:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829719.445431:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518138368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829719.445438:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518138496:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829719.445460:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829719.445472:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff8b2e88075680 x1874672518138496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829719.445493:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518138496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829719.445503:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829719.445515:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829719.445519:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518138496:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829719.445526:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518138624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829719.445547:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829719.445560:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff8b2e88077c00 x1874672518138624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829719.445573:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518138624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829719.445582:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829719.445585:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829719.445589:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518138624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829719.445595:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518138752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829719.445615:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829719.445625:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829719/real 1787829719] req@ffff8b2e88074780 x1874672518138752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829719.445638:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518138752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829744 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829719.445690:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829719.445695:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829719.445698:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518138752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829719.764771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829720.276769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829720.276874:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829720.468712:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829720.468776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829720.788730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829721.300758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829721.300835:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829721.492754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829721.492754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829721.812740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829722.324763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829722.324868:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829722.516788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829722.516796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829722.836811:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829723.348761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829723.348884:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829723.540713:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829723.540768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829723.860770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829724.372759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829724.372834:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829724.564778:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829724.564791:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829724.564792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829724.564792:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829724.564799:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829724.564812:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2730 00000100:00080000:1.0:1787829724.564822:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829724.564830:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829724.564874:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518138880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829724.564898:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829724.564900:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829724.564902:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829724.564905:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2730 00000100:00080000:1.0:1787829724.564909:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829724.564913:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829724.564922:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829724.564931:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518138880:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829724.564936:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518139008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829724.564946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829724.564949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829724.564950:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829724.564953:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2730 00000100:00080000:1.0:1787829724.564955:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829724.564957:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829724.564960:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829724.564968:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518139136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829724.564975:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829724.564978:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829724.564979:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829724.564981:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2730 00000100:00080000:1.0:1787829724.564984:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829724.564986:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829724.564989:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829724.564996:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518139264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829724.565002:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829724.565006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829724.565009:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829724.565014:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2730 00000100:00080000:1.0:1787829724.565019:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829724.565023:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829724.565040:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829724.565045:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518139008:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829724.565060:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518139392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829724.565075:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518139136:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829724.565075:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829724.565083:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829724.565085:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829724.565090:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2730 00000100:00080000:1.0:1787829724.565095:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829724.565099:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518139264:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829724.565100:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829724.565109:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829724.565121:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518139520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829724.565129:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829724.565131:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829724.565133:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829724.565135:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2730 00000100:00080000:1.0:1787829724.565139:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829724.565140:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518139392:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829724.565143:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829724.565151:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829724.565161:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518139648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829724.565183:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518139520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829724.565214:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518139648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829724.884810:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829725.396760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829725.396838:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829725.588759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829725.588767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829725.908746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829726.420767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829726.420863:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829726.612733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829726.612754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829726.932703:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829727.444761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829727.444827:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829727.636806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829727.636807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:0.0:1787829727.636996:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:0.0:1787829727.637062:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:0.0:1787829727.637082:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829727.637131:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829727.637144:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829727.637157:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829727.637166:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829727.637179:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:0.0:1787829727.637186:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829727.637370:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829727] req@ffff8b2e88075e00 x1874672518139648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829727.637396:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518139648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829727.637423:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829727.637428:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829727.637433:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518139648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829727.637441:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829727] req@ffff8b2e88074000 x1874672518139392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829727.637456:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518139392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829727.637464:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829727.637469:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829727.637473:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518139392:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829727.637493:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829727] req@ffff8b2e880761c0 x1874672518139520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829727.637508:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518139520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829727.637517:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829727.637520:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829727.637524:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518139520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829727.637531:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829727] req@ffff8b2e88076580 x1874672518139008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829727.637542:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518139008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829727.637551:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829727.637554:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829727.637558:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518139008:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829727.637563:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829727] req@ffff8b2e88076d00 x1874672518139136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829727.637576:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518139136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829727.637584:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829727.637589:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829727.637592:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518139136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829727.637599:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829727] req@ffff8b2e880770c0 x1874672518139264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829727.637619:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518139264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829727.637627:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829727.637630:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829727.637633:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518139264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829727.637639:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829724/real 1787829727] req@ffff8b2e88077840 x1874672518138880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829727.637687:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518138880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829749 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829727.637700:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829727.637705:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829727.637709:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518138880:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829727.956731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829728.468738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829728.468808:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829728.660757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829728.660793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829728.980789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829729.492769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829729.492810:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829729.684773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829729.684778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829729.684794:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829729.684801:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829729.684807:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.684817:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2736 00000100:00080000:3.0:1787829729.684830:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.684841:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829729.684891:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518139776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829729.684924:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829729.684928:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829729.684930:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.684936:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2736 00000100:00080000:3.0:1787829729.684940:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.684945:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829729.684958:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829729.684977:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518139776:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829729.684977:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518139904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829729.685021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829729.685025:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829729.685027:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.685041:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2736 00000100:00080000:3.0:1787829729.685045:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.685049:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829729.685055:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829729.685068:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518140032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829729.685078:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829729.685082:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787829729.685087:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829729.685087:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.685097:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2736 00000100:00080000:3.0:1787829729.685101:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.685106:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829729.685112:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829729.685125:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518140160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829729.685134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829729.685138:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829729.685140:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.685145:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2736 00000100:00080000:3.0:1787829729.685149:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.685153:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829729.685159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829729.685161:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff8b2e8803b840 x1874672518139776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829729.685175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518140288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829729.685186:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829729.685188:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518139776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829729.685192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829729.685197:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.685207:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2736 00000100:00080000:0.0:1787829729.685212:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829729.685213:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.685223:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829729.685224:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829729.685232:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518139776:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787829729.685232:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829729.685251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518140416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829729.685264:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829729.685268:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518139904:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829729.685270:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829729.685274:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829729.685282:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2736 00000100:00080000:3.0:1787829729.685286:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829729.685290:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829729.685296:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829729.685308:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829729.685312:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518140544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829729.685330:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff8b2e8803b0c0 x1874672518139904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829729.685346:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518139904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829729.685357:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829729.685362:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829729.685366:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518139904:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829729.685373:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518140032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829729.685398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829729.685411:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff8b2e8803a1c0 x1874672518140032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829729.685425:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518140032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829729.685435:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829729.685439:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829729.685443:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518140032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829729.685457:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518140160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829729.685482:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829729.685495:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff8b2e88038f00 x1874672518140160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829729.685508:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518140160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829729.685516:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829729.685520:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829729.685523:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518140160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829729.685530:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518140288:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829729.685553:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829729.685565:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff8b2e8803a580 x1874672518140288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829729.685579:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518140288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829729.685588:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829729.685591:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829729.685594:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518140288:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829729.685613:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518140416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829729.685638:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829729.685679:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff8b2e880383c0 x1874672518140416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829729.685694:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518140416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829729.685704:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829729.685708:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829729.685711:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518140416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829729.685718:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518140544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829729.685744:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829729.685756:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829729/real 1787829729] req@ffff8b2e880392c0 x1874672518140544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829729.685776:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518140544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829754 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829729.685784:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829729.685787:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829729.685790:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518140544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829730.004738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829730.516764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829730.516842:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829730.708731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829730.708734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829731.028767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829731.540788:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829731.540827:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829731.732755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829731.732755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829732.052719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829732.564770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829732.564874:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829732.756776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829732.756791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829733.076777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829733.588765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829733.588836:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829733.780770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829733.780770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829734.100761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829734.612769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829734.612823:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829734.804749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829734.804755:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829734.804763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829734.804770:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829734.804778:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829734.804791:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2741 00000100:00080000:3.0:1787829734.804806:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829734.804818:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829734.804867:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518140672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829734.804892:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829734.804896:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829734.804897:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829734.804901:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2741 00000100:00080000:3.0:1787829734.804904:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829734.804908:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829734.804919:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829734.804927:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518140672:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829734.804932:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518140800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829734.804945:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829734.804949:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829734.804951:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829734.804967:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2741 00000100:00080000:3.0:1787829734.804972:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829734.804976:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829734.804983:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829734.804999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518140928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829734.805009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829734.805014:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829734.805015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829734.805020:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829734.805025:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2741 00000100:00080000:3.0:1787829734.805030:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829734.805036:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829734.805042:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829734.805058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518141056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829734.805062:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff8b2e88039a40 x1874672518140672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829734.805077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829734.805081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829734.805083:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829734.805087:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518140672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829734.805089:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2741 00000100:00080000:3.0:1787829734.805096:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829734.805102:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829734.805103:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829734.805108:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829734.805109:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829734.805114:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518140672:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829734.805123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518141184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829734.805130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829734.805133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829734.805134:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829734.805139:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518140800:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829734.805139:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2741 00000100:00080000:3.0:1787829734.805144:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829734.805148:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829734.805151:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829734.805162:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518141312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829734.805168:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829734.805171:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829734.805175:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829734.805177:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829734.805182:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2741 00000100:00100000:0.0:1787829734.805184:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff8b2e88039e00 x1874672518140800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829734.805186:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829734.805194:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829734.805202:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829734.805206:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518140800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829734.805214:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518141440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829734.805215:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.805223:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829734.805225:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518140800:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829734.805247:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518140928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829734.805263:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829734.805271:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff8b2e8803b480 x1874672518140928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829734.805283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518140928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829734.805289:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.805292:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829734.805294:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518140928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829734.805297:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518141056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829734.805308:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829734.805314:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff8b2e88039680 x1874672518141056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829734.805321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518141056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829734.805331:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.805333:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829734.805336:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518141056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829734.805339:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518141184:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829734.805349:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829734.805354:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff8b2e880392c0 x1874672518141184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829734.805361:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518141184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829734.805367:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.805369:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829734.805371:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518141184:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829734.805382:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518141312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829734.805392:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829734.805398:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff8b2e880383c0 x1874672518141312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829734.805406:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518141312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829734.805412:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.805415:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829734.805419:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518141312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829734.805426:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518141440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829734.805445:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829734.805456:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829734/real 1787829734] req@ffff8b2e8803a580 x1874672518141440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829734.805470:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518141440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829759 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829734.805476:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829734.805478:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829734.805480:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518141440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829735.124734:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829735.636793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829735.636858:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829735.828761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829735.828761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829736.148721:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829736.660736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829736.660801:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829736.852747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829736.852762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829737.172719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829737.684758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829737.684845:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829737.876746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829737.876862:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829738.196716:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829738.708727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829738.708803:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829738.900729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829738.900755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829739.220754:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829739.732755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829739.732824:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829739.924758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829739.924772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829739.924773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829739.924783:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829739.924790:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.924803:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2746 00000100:00080000:3.0:1787829739.924815:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.924825:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829739.924869:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518141568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829739.924895:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829739.924899:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829739.924900:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.924904:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2746 00000100:00080000:3.0:1787829739.924908:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.924911:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829739.924941:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829739.924946:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518141568:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829739.924959:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518141696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829739.924985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829739.924989:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829739.924991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.924994:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2746 00000100:00080000:3.0:1787829739.924998:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.925003:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829739.925008:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829739.925018:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518141824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829739.925026:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829739.925028:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829739.925030:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.925032:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2746 00000100:00080000:3.0:1787829739.925034:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829739.925039:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829739.925040:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829739.925046:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829739.925055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518141952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829739.925062:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829739.925065:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829739.925066:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.925069:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2746 00000100:00080000:3.0:1787829739.925071:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.925074:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829739.925077:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829739.925086:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff8b2e88038f00 x1874672518141568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829739.925088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518142080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829739.925112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829739.925116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829739.925119:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.925122:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2746 00000100:00080000:3.0:1787829739.925124:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829739.925127:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518141568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829739.925127:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829739.925134:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829739.925145:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829739.925147:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518142208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829739.925151:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829739.925176:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829739.925182:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518141568:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829739.925182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829739.925187:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829739.925194:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2746 00000100:00080000:3.0:1787829739.925198:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829739.925201:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829739.925205:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829739.925214:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518142336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829739.925218:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518141696:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829739.925270:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829739.925283:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff8b2e8803a1c0 x1874672518141696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829739.925293:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518141696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.925300:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829739.925303:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829739.925305:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518141696:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829739.925310:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518141824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829739.925324:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829739.925330:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff8b2e8803b0c0 x1874672518141824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829739.925339:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518141824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.925345:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829739.925347:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829739.925349:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518141824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829739.925353:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518141952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829739.925372:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829739.925380:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff8b2e8803b840 x1874672518141952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829739.925389:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518141952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.925396:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829739.925398:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829739.925400:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518141952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829739.925404:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518142080:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829739.925416:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829739.925422:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff8b2e8803a580 x1874672518142080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829739.925431:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518142080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.925437:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829739.925439:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829739.925442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518142080:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829739.925449:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518142208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829739.925469:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829739.925500:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff8b2e880383c0 x1874672518142208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829739.925515:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518142208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.925527:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829739.925531:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829739.925534:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518142208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829739.925566:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518142336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829739.925585:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829739.925593:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829739/real 1787829739] req@ffff8b2e880392c0 x1874672518142336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829739.925602:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518142336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829764 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829739.925609:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829739.925611:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829739.925613:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518142336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829740.244720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829740.756728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829740.756801:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829740.948736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829740.948777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829741.268723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829741.780778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829741.780859:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829741.972752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829741.972752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829742.292720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829742.804777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829742.804856:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829742.996725:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829742.996781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829743.316762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829743.828801:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829743.828912:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829744.020759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829744.020759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829744.340739:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829744.852780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829744.852816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829745.044761:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829745.044764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829745.044768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829745.044777:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829745.044785:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829745.044797:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2751 00000100:00080000:3.0:1787829745.044811:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829745.044819:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829745.044855:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518142464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829745.044879:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829745.044898:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829745.044900:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829745.044904:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2751 00000100:00080000:3.0:1787829745.044907:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829745.044912:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829745.044921:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829745.044934:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518142464:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829745.044938:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518142592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829745.044954:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829745.044959:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829745.044961:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829745.044967:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2751 00000100:00080000:3.0:1787829745.044972:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829745.044977:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829745.044984:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1787829745.044994:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829745.044999:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518142720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829745.045009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829745.045013:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829745.045016:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829745.045020:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2751 00000100:00080000:3.0:1787829745.045026:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829745.045031:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829745.045038:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829745.045041:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829745/real 1787829745] req@ffff8b2e88039680 x1874672518142464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829745.045055:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518142848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829745.045059:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518142464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829745.045105:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829745.045107:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829745.045109:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829745.045115:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787829745.045116:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518142464:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829745.045121:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829745.045129:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2751 00000100:00080000:3.0:1787829745.045134:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829745.045139:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829745.045141:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518142592:192.168.203.101@tcp:38:kworker.0 00000008:00100000:3.0:1787829745.045149:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829745.045167:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829745.045169:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518142976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829745.045180:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829745/real 1787829745] req@ffff8b2e8803b480 x1874672518142592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829745.045186:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829745.045193:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518142592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829745.045201:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829745.045201:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829745.045204:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829745.045207:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829745.045213:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2751 00000100:00100000:0.0:1787829745.045214:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518142592:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829745.045221:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829745.045225:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518142720:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787829745.045229:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829745.045237:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829745.045243:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829745.045254:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829745/real 1787829745] req@ffff8b2e88039e00 x1874672518142720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829745.045256:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518143104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829745.045270:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518142720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829745.045295:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829745.045300:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829745.045301:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829745.045306:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518142720:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787829745.045307:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829745.045311:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829745.045315:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518142848:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787829745.045316:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2751 00000100:00080000:3.0:1787829745.045324:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829745.045329:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829745.045334:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829745.045337:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829745.045346:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829745/real 1787829745] req@ffff8b2e88039a40 x1874672518142848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829745.045350:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518143232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829745.045366:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518142848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829745.045373:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829745.045375:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829745.045378:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518142848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829745.045390:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518142976:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829745.045409:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829745.045417:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829745/real 1787829745] req@ffff8b2e88039680 x1874672518142976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829745.045424:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518142976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829745.045430:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829745.045432:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829745.045434:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518142976:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829745.045438:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518143104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829745.045448:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829745.045454:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829745/real 1787829745] req@ffff8b2e880392c0 x1874672518143104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829745.045462:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518143104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829745.045467:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829745.045469:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829745.045470:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518143104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829745.045474:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518143232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829745.045483:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829745.045489:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829745/real 1787829745] req@ffff8b2e880383c0 x1874672518143232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829745.045496:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518143232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829770 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829745.045501:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829745.045502:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829745.045504:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518143232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829745.364760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829745.876786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829745.876842:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829746.068757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829746.068761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829746.388784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829746.900797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829746.900863:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829747.092722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829747.092788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829747.412720:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829747.924805:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829747.924895:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829748.116793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829748.116795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829748.436782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829748.948739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829748.948817:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829749.140807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829749.140809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829749.460770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829749.972751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829749.972868:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829750.164741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829750.164760:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829750.164768:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829750.164774:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829750.164778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829750.164787:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2756 00000100:00080000:3.0:1787829750.164802:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829750.164813:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829750.164881:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518143360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829750.164910:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829750.164914:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829750.164916:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829750.164921:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2756 00000100:00080000:3.0:1787829750.164925:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829750.164928:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829750.164941:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829750.164977:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518143360:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829750.164977:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518143488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829750.164998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829750.165002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829750.165003:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829750.165007:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2756 00000100:00080000:3.0:1787829750.165011:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829750.165015:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829750.165019:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829750.165032:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518143616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829750.165042:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829750.165046:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829750.165048:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829750.165051:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2756 00000100:00080000:3.0:1787829750.165054:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829750.165057:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829750.165062:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829750.165073:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518143744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829750.165083:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829750.165091:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829750.165096:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829750.165099:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829750.165105:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2756 00000100:00080000:3.0:1787829750.165109:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829750.165114:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829750.165118:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829750.165128:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518143872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829750.165139:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829750.165142:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829750.165144:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829750.165148:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829750/real 1787829750] req@ffff8b2e8803a580 x1874672518143360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829750.165151:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2756 00000100:00080000:3.0:1787829750.165156:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829750.165159:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829750.165163:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829750.165176:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518144000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829750.165186:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829750.165190:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829750.165193:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829750.165197:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518143360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829750.165200:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2756 00000100:00080000:3.0:1787829750.165207:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829750.165216:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829750.165225:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829750.165229:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829750.165237:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829750.165240:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518144128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829750.165248:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518143360:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829750.165287:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518143488:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829750.165331:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829750.165344:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829750/real 1787829750] req@ffff8b2e8803b840 x1874672518143488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829750.165356:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518143488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829750.165365:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829750.165368:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829750.165372:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518143488:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829750.165380:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518143616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829750.165409:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829750.165425:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829750/real 1787829750] req@ffff8b2e8803b0c0 x1874672518143616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829750.165443:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518143616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829750.165463:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829750.165466:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829750.165469:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518143616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829750.165475:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518143744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829750.165496:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829750.165506:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829750/real 1787829750] req@ffff8b2e8803a1c0 x1874672518143744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829750.165516:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518143744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829750.165524:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829750.165527:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829750.165530:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518143744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829750.165534:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518143872:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829750.165548:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829750.165556:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829750/real 1787829750] req@ffff8b2e88038f00 x1874672518143872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829750.165567:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518143872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829750.165579:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829750.165584:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829750.165590:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518143872:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829750.165615:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518144000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829750.165639:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829750.165653:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829750/real 1787829750] req@ffff8b2e880383c0 x1874672518144000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829750.165664:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518144000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829750.165670:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829750.165673:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829750.165726:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518144000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829750.165737:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518144128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829750.165770:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829750.165801:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829750/real 1787829750] req@ffff8b2e880392c0 x1874672518144128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829750.165811:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518144128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829775 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829750.165818:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829750.165821:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829750.165823:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518144128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829750.484752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829750.996746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829750.996839:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829751.188795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829751.188800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829751.508748:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829752.020743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829752.020870:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829752.212726:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829752.212824:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829752.532721:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829753.044776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829753.044785:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829753.236770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829753.236770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829753.556768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829754.068748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829754.068874:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829754.260802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829754.260802:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829754.580797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829755.092804:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829755.092875:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829755.284771:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829755.284781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829755.284784:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787829755.284786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829755.284791:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.284822:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2761 00000100:00080000:3.0:1787829755.284839:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.284850:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829755.284894:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518144256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829755.284917:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.284921:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829755.284922:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.284926:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2761 00000100:00080000:3.0:1787829755.284930:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.284934:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829755.284944:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829755.284950:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518144256:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829755.284956:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518144384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829755.284966:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.284968:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829755.284969:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.284972:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2761 00000100:00080000:3.0:1787829755.284974:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.284977:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829755.284981:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829755.284991:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518144512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829755.284999:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.285002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829755.285003:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.285005:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2761 00000100:00080000:3.0:1787829755.285007:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.285010:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829755.285013:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829755.285022:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518144640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829755.285026:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829755.285034:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.285036:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829755.285038:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.285042:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2761 00000100:00080000:3.0:1787829755.285045:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.285049:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829755.285052:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829755.285060:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518144768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829755.285066:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.285070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829755.285071:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829755.285074:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff8b2e88039680 x1874672518144256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829755.285077:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2761 00000100:00080000:3.0:1787829755.285090:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829755.285093:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518144256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829755.285097:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829755.285104:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829755.285110:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829755.285116:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829755.285116:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518144896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829755.285123:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518144256:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829755.285132:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829755.285135:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829755.285136:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829755.285140:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2761 00000100:00080000:3.0:1787829755.285142:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829755.285145:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829755.285148:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829755.285155:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518145024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829755.285160:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518144384:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829755.285193:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829755.285205:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff8b2e88039a40 x1874672518144384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829755.285215:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518144384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829755.285221:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829755.285224:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829755.285226:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518144384:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829755.285231:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518144512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829755.285246:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829755.285253:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff8b2e88039e00 x1874672518144512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829755.285261:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518144512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829755.285266:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829755.285268:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829755.285270:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518144512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829755.285274:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518144640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829755.285284:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829755.285301:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff8b2e8803b480 x1874672518144640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829755.285310:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518144640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829755.285319:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829755.285323:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829755.285326:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518144640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829755.285332:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518144768:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829755.285355:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829755.285367:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff8b2e880392c0 x1874672518144768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829755.285388:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518144768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829755.285398:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829755.285402:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829755.285405:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518144768:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829755.285413:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518144896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829755.285430:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829755.285438:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff8b2e880383c0 x1874672518144896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829755.285447:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518144896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829755.285452:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829755.285454:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829755.285456:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518144896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829755.285469:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518145024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829755.285481:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829755.285491:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829755/real 1787829755] req@ffff8b2e88038f00 x1874672518145024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829755.285503:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518145024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829780 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829755.285519:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829755.285523:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829755.285536:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518145024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829755.604731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829756.116743:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829756.116845:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829756.308775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829756.308778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829756.628775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829757.140751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829757.140846:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829757.332743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829757.332752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829757.652746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829758.164714:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829758.164835:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829758.356736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829758.356768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829758.676728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829759.188770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829759.188858:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829759.380773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829759.380803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829759.700696:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829760.212764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829760.212867:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829760.404751:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829760.404760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829760.404761:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829760.404767:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829760.404769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829760.404792:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2766 00000100:00080000:3.0:1787829760.404806:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829760.404816:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829760.404865:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518145152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829760.404895:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829760.404898:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829760.404900:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829760.404903:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2766 00000100:00080000:3.0:1787829760.404906:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829760.404910:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829760.404920:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829760.404932:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518145280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829760.404941:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829760.404946:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829760.404947:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518145152:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829760.404948:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829760.404955:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2766 00000100:00080000:3.0:1787829760.404959:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829760.404961:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829760.404965:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829760.404972:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518145408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829760.404979:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829760.404981:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829760.404982:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829760.404985:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2766 00000100:00080000:3.0:1787829760.404988:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829760.404991:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829760.404994:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829760.405002:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518145536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829760.405010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829760.405014:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829760.405016:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829760.405021:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2766 00000100:00080000:3.0:1787829760.405025:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829760.405029:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829760.405034:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829760.405041:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829760.405049:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518145664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829760.405060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829760.405064:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829760.405066:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829760.405070:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2766 00000100:00080000:3.0:1787829760.405074:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829760.405078:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829760.405083:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829760.405101:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff8b2e8803a1c0 x1874672518145152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829760.405102:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518145792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829760.405141:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829760.405143:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829760.405145:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829760.405148:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2766 00000100:00080000:3.0:1787829760.405152:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829760.405153:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518145152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829760.405158:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829760.405165:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829760.405179:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829760.405179:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518145920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829760.405190:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829760.405199:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518145152:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829760.405228:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518145280:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829760.405257:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829760.405277:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff8b2e8803b0c0 x1874672518145280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829760.405292:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518145280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829760.405304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829760.405308:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829760.405312:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518145280:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829760.405319:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518145408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829760.405342:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829760.405356:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff8b2e8803b840 x1874672518145408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829760.405369:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518145408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829760.405379:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829760.405383:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829760.405387:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518145408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829760.405395:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518145536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829760.405417:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829760.405430:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff8b2e8803a580 x1874672518145536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829760.405443:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518145536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829760.405452:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829760.405456:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829760.405460:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518145536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829760.405467:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518145664:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829760.405488:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829760.405501:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff8b2e88038f00 x1874672518145664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829760.405516:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518145664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829760.405526:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829760.405530:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829760.405534:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518145664:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829760.405540:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518145792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829760.405562:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829760.405574:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff8b2e880383c0 x1874672518145792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829760.405587:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518145792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829760.405595:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829760.405598:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829760.405601:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518145792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829760.405608:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518145920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829760.405628:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829760.405640:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829760/real 1787829760] req@ffff8b2e880392c0 x1874672518145920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829760.405679:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518145920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829785 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829760.405689:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829760.405693:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829760.405697:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518145920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829760.724807:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829761.236760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829761.236826:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829761.428755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829761.428856:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829761.748696:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829762.260792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829762.260989:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829762.452739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829762.452747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829762.772755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829763.284772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829763.284839:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829763.476746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829763.476746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829763.796808:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829764.308762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829764.308877:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829764.500709:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829764.500766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829764.820803:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829765.332735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829765.332898:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829765.524739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829765.524739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829765.524779:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829765.524791:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829765.524797:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829765.524810:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2771 00000100:00080000:3.0:1787829765.524825:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829765.524834:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829765.524886:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518146048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829765.524911:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829765.524915:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829765.524916:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829765.524920:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2771 00000100:00080000:3.0:1787829765.524923:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829765.524927:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829765.524937:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829765.524952:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518146048:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829765.524953:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518146176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829765.524977:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829765.524980:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829765.524981:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829765.524985:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2771 00000100:00080000:3.0:1787829765.524987:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829765.524990:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829765.524993:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829765.525009:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518146304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829765.525016:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829765.525019:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829765.525021:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829765.525024:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2771 00000100:00080000:3.0:1787829765.525027:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829765.525030:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829765.525035:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829765.525041:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829765.525045:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518146432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829765.525055:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829765.525058:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829765.525060:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829765.525066:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2771 00000100:00080000:3.0:1787829765.525069:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829765.525073:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829765.525077:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829765.525088:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518146560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829765.525095:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829765] req@ffff8b2e8803b480 x1874672518146048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829765.525106:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829765.525122:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829765.525129:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829765.525135:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518146048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829765.525137:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2771 00000100:00080000:3.0:1787829765.525146:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829765.525159:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829765.525164:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829765.525166:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829765.525174:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829765.525182:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518146048:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829765.525187:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518146688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829765.525211:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829765.525215:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829765.525218:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829765.525223:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2771 00000100:00080000:3.0:1787829765.525227:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829765.525231:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518146176:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829765.525234:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829765.525245:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829765.525260:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829765.525269:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518146816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829765.525278:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829765] req@ffff8b2e88039e00 x1874672518146176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829765.525307:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518146176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829765.525318:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829765.525321:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829765.525324:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518146176:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829765.525329:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518146304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829765.525343:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829765.525349:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829765] req@ffff8b2e88039a40 x1874672518146304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829765.525357:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518146304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829765.525362:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829765.525364:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829765.525366:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518146304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829765.525372:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518146432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829765.525390:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829765.525401:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829765] req@ffff8b2e88039680 x1874672518146432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829765.525413:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518146432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829765.525423:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829765.525426:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829765.525429:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518146432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829765.525434:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518146560:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829765.525449:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829765.525456:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829765] req@ffff8b2e880392c0 x1874672518146560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829765.525464:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518146560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829765.525469:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829765.525471:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829765.525473:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518146560:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829765.525477:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518146688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829765.525487:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829765.525493:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829765] req@ffff8b2e880383c0 x1874672518146688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829765.525499:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518146688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829765.525505:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829765.525506:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829765.525509:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518146688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829765.525524:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518146816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829765.525534:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829765.525540:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829765/real 1787829765] req@ffff8b2e88038f00 x1874672518146816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829765.525547:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518146816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829790 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829765.525552:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829765.525554:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829765.525556:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518146816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829765.844790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829766.356789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829766.356883:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829766.548796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829766.548803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829766.868762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829767.380750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829767.380851:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829767.572795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829767.572803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829767.892784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829768.404735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829768.404886:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829768.596780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829768.596780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829768.916786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829769.428770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829769.428877:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829769.620775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829769.620775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829769.940696:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829770.452700:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829770.452809:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829770.644823:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829770.644823:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829770.644828:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829770.644843:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829770.644850:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829770.644864:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2777 00000100:00080000:3.0:1787829770.644880:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829770.644891:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829770.644949:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518146944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829770.644984:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829770.644988:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829770.644991:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829770.644997:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2777 00000100:00080000:3.0:1787829770.645001:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829770.645007:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829770.645019:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829770.645037:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518146944:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829770.645037:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518147072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829770.645056:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829770.645060:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829770.645062:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829770.645067:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2777 00000100:00080000:3.0:1787829770.645071:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829770.645076:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829770.645083:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829770.645097:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518147200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829770.645108:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829770.645112:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829770.645115:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829770.645118:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2777 00000100:00080000:3.0:1787829770.645122:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829770.645129:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787829770.645130:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:3.0:1787829770.645136:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829770.645159:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518147328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829770.645170:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829770.645174:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829770.645176:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829770.645181:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2777 00000100:00080000:3.0:1787829770.645186:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829770.645189:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff8b2e8803a580 x1874672518146944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829770.645194:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829770.645202:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829770.645218:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518147456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829770.645228:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518146944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829770.645233:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829770.645242:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829770.645246:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829770.645252:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829770.645254:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2777 00000100:00080000:0.0:1787829770.645262:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829770.645270:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518146944:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829770.645272:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829770.645282:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829770.645289:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829770.645303:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518147584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829770.645307:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518147072:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829770.645321:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829770.645329:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829770.645331:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829770.645336:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2777 00000100:00080000:3.0:1787829770.645340:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829770.645346:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829770.645352:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829770.645358:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829770.645371:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518147712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829770.645379:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff8b2e8803b840 x1874672518147072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.645408:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518147072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.645418:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.645423:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829770.645426:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518147072:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829770.645433:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518147200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829770.645465:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829770.645478:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff8b2e8803b0c0 x1874672518147200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.645491:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518147200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.645500:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.645503:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829770.645515:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518147200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829770.645522:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518147328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829770.645542:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829770.645554:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff8b2e8803a1c0 x1874672518147328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.645566:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518147328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.645575:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.645578:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829770.645582:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518147328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829770.645588:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518147456:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829770.645604:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829770.645614:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff8b2e88038f00 x1874672518147456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.645625:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518147456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.645635:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.645638:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829770.645642:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518147456:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829770.645681:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518147584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829770.645700:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829770.645727:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff8b2e880383c0 x1874672518147584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.645738:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518147584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.645747:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.645751:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829770.645754:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518147584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829770.645760:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518147712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829770.645777:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829770.645787:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829770/real 1787829770] req@ffff8b2e880392c0 x1874672518147712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829770.645798:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518147712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829795 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829770.645805:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829770.645809:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829770.645812:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518147712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829770.964776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829771.476747:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829771.476804:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829771.668734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829771.668740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829771.988782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829772.500777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829772.500862:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829772.692826:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829772.692830:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829773.012807:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829773.524733:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829773.524823:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829773.716784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829773.716784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829774.036796:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829774.548814:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829774.548874:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829774.740778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829774.740779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829775.060795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829775.572741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829775.572813:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829775.764744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829775.764744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829775.764752:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829775.764758:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829775.764762:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.764770:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2782 00000100:00080000:2.0:1787829775.764780:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.764787:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829775.764820:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518147840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829775.764843:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829775.764846:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829775.764848:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.764859:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2782 00000100:00080000:2.0:1787829775.764863:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.764867:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829775.764875:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518147840:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787829775.764876:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829775.764890:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518147968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829775.764902:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829775.764905:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829775.764907:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.764910:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2782 00000100:00080000:2.0:1787829775.764914:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.764917:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829775.764922:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1787829775.764927:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829775.764934:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518148096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829775.764944:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829775.764947:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829775.764949:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.764952:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2782 00000100:00080000:2.0:1787829775.764955:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.764958:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829775.764962:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829775.764969:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff8b2e8801da40 x1874672518147840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829775.764973:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518148224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829775.764995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829775.764999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787829775.765000:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518147840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829775.765002:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.765012:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2782 00000100:00080000:0.0:1787829775.765014:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829775.765015:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829775.765019:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787829775.765022:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829775.765025:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829775.765031:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518147840:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829775.765037:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518148352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829775.765047:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829775.765051:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518147968:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829775.765052:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829775.765057:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.765063:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2782 00000100:00080000:2.0:1787829775.765066:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.765069:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829775.765073:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829775.765082:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829775.765086:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518148480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829775.765097:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829775.765100:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff8b2e8801fc00 x1874672518147968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829775.765101:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829775.765104:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829775.765111:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2782 00000100:00080000:2.0:1787829775.765116:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829775.765120:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829775.765124:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518147968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787829775.765127:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829775.765136:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829775.765139:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829775.765142:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518148608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829775.765143:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518147968:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829775.765160:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518148096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829775.765177:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829775.765184:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff8b2e8801f480 x1874672518148096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829775.765191:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518148096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829775.765196:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829775.765198:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829775.765205:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518148096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829775.765209:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518148224:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829775.765219:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829775.765224:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff8b2e8801cf00 x1874672518148224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829775.765231:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518148224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829775.765235:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829775.765238:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829775.765239:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518148224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829775.765242:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518148352:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829775.765250:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829775.765255:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff8b2e8801c000 x1874672518148352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829775.765261:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518148352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829775.765266:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829775.765267:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829775.765269:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518148352:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829775.765279:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518148480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829775.765288:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829775.765293:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff8b2e8801e580 x1874672518148480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829775.765300:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518148480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829775.765304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829775.765306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829775.765307:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518148480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829775.765310:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518148608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829775.765318:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829775.765323:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829775/real 1787829775] req@ffff8b2e8801e940 x1874672518148608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829775.765329:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518148608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829800 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829775.765333:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829775.765335:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829775.765337:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518148608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829776.084782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829776.596791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829776.596926:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829776.788753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829776.788754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829777.108722:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829777.620771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829777.620838:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829777.812758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829777.812780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829778.132771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829778.644774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829778.644836:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829778.836789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829778.836789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829779.156771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829779.668724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829779.668830:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829779.860766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829779.860766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829780.180746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829780.692762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829780.692836:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829780.884748:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829780.884754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829780.884756:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1787829780.884762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829780.884763:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.884774:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2787 00000100:00080000:3.0:1787829780.884785:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.884799:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829780.884842:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518148736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829780.884862:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.884865:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829780.884866:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.884869:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2787 00000100:00080000:3.0:1787829780.884871:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.884873:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829780.884882:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829780.884891:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518148864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829780.884898:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.884902:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829780.884904:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829780.884905:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518148736:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829780.884907:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2787 00000100:00080000:3.0:1787829780.884911:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.884916:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829780.884918:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829780.884927:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518148992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829780.884936:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.884940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829780.884942:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.884945:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2787 00000100:00080000:3.0:1787829780.884949:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.884954:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829780.884958:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829780.884970:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518149120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829780.884980:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829780.884981:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829780.884993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829780.884996:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.885001:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2787 00000100:00080000:3.0:1787829780.885005:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.885010:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829780.885015:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829780.885020:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff8b2e88039680 x1874672518148736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.885034:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518148736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.885043:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829780.885043:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518149248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829780.885051:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829780.885058:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518148736:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829780.885060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.885062:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829780.885064:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.885068:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2787 00000100:00080000:3.0:1787829780.885073:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.885078:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829780.885084:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829780.885097:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518149376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829780.885098:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518148864:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829780.885128:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829780.885130:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829780.885132:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829780.885136:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2787 00000800:00000100:0.0:1787829780.885141:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829780.885142:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829780.885148:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829780.885152:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829780.885164:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518149504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829780.885176:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff8b2e88039a40 x1874672518148864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.885193:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518148864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.885203:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.885207:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829780.885210:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518148864:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829780.885215:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518148992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829780.885233:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829780.885240:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff8b2e88039e00 x1874672518148992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.885248:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518148992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.885257:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.885259:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829780.885262:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518148992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829780.885267:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518149120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829780.885284:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829780.885295:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff8b2e8803b480 x1874672518149120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.885307:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518149120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.885315:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.885317:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829780.885319:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518149120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829780.885323:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518149248:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829780.885338:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829780.885344:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff8b2e880392c0 x1874672518149248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.885352:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518149248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.885357:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.885359:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829780.885361:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518149248:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829780.885372:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518149376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829780.885382:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829780.885388:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff8b2e880383c0 x1874672518149376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.885395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518149376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.885400:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.885401:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829780.885403:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518149376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829780.885408:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518149504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829780.885424:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829780.885443:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829780/real 1787829780] req@ffff8b2e88038f00 x1874672518149504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829780.885455:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518149504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829805 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829780.885462:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829780.885465:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829780.885469:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518149504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829781.204705:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829781.716750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829781.716822:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829781.908737:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829781.908787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829782.228731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829782.740759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829782.740822:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829782.932765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829782.932765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829783.252773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829783.764772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829783.764847:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829783.956718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829783.956765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829784.276757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829784.788740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829784.788806:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829784.980738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829784.980738:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829785.300756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829785.812718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829785.812818:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829786.004715:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829786.004723:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829786.004732:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829786.004738:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829786.004740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829786.004751:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2792 00000100:00080000:3.0:1787829786.004766:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829786.004778:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829786.004823:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518149632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829786.004845:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829786.004848:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829786.004849:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829786.004853:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2792 00000100:00080000:3.0:1787829786.004857:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829786.004860:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829786.004870:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829786.004875:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518149632:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829786.004882:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518149760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829786.004890:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829786.004892:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829786.004893:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829786.004896:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2792 00000100:00080000:3.0:1787829786.004898:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829786.004900:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829786.004903:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829786.004910:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518149888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829786.004916:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829786.004919:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829786.004920:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829786.004922:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2792 00000100:00080000:3.0:1787829786.004924:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829786.004926:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829786.004928:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829786.004935:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518150016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829786.004940:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829786.004942:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829786.004945:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829786.004946:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829786.004958:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2792 00000100:00080000:3.0:1787829786.004962:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829786.004965:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829786.004969:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829786.004977:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518150144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829786.004983:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829786.004986:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829786.004987:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829786.004991:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829786/real 1787829786] req@ffff8b2e8803a1c0 x1874672518149632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829786.004993:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2792 00000100:00080000:3.0:1787829786.004997:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829786.005002:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829786.005008:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829786.005015:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518149632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829786.005017:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518150272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829786.005042:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829786.005044:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829786.005050:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829786.005050:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829786.005052:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829786.005057:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518149632:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829786.005059:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2792 00000100:00080000:3.0:1787829786.005066:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829786.005068:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829786.005071:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829786.005078:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518150400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829786.005085:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518149760:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829786.005105:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829786.005117:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829786/real 1787829786] req@ffff8b2e8803b0c0 x1874672518149760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829786.005128:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518149760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829786.005144:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829786.005147:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829786.005151:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518149760:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829786.005157:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518149888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829786.005177:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829786.005187:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829786/real 1787829786] req@ffff8b2e8803b840 x1874672518149888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829786.005199:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518149888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829786.005207:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829786.005210:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829786.005214:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518149888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829786.005220:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518150016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829786.005237:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829786.005244:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829786/real 1787829786] req@ffff8b2e8803a580 x1874672518150016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829786.005251:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518150016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829786.005256:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829786.005258:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829786.005260:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518150016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829786.005263:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518150144:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829786.005272:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829786.005277:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829786/real 1787829786] req@ffff8b2e88038f00 x1874672518150144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829786.005288:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518150144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829786.005293:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829786.005295:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829786.005296:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518150144:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829786.005299:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518150272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829786.005308:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829786.005316:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829786/real 1787829786] req@ffff8b2e880383c0 x1874672518150272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829786.005327:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518150272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829786.005335:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829786.005338:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829786.005341:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518150272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829786.005365:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518150400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829786.005381:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829786.005388:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829786/real 1787829786] req@ffff8b2e880392c0 x1874672518150400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829786.005395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518150400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829811 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829786.005404:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829786.005406:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829786.005408:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518150400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829786.324728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829786.836782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829786.836830:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829787.028766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829787.028766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829787.348757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829787.860760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829787.860821:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829788.052744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829788.052761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829788.372767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829788.884740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829788.884790:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829789.076768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829789.076799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829789.396745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829789.908731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829789.908813:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829790.100748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829790.100748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829790.420749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829790.932777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829790.932863:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829791.124748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829791.124758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829791.124797:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829791.124804:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829791.124809:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829791.124822:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2797 00000100:00080000:1.0:1787829791.124836:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829791.124845:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829791.124896:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518150528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829791.124924:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829791.124931:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829791.124934:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829791.124941:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2797 00000100:00080000:1.0:1787829791.124946:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829791.124955:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518150528:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829791.124963:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829791.124975:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829791.124990:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518150656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829791.124999:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829791.125002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829791.125004:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829791.125007:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2797 00000100:00080000:1.0:1787829791.125022:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829791.125025:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829791.125029:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829791.125038:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518150784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829791.125045:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829791.125048:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829791.125050:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829791.125052:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2797 00000100:00080000:1.0:1787829791.125056:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829791.125059:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829791.125063:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829791.125069:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518150656:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829791.125075:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518150912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829791.125084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829791.125087:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829791.125090:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829791.125094:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2797 00000100:00100000:0.0:1787829791.125098:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518150784:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829791.125098:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829791.125108:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829791.125116:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829791.125133:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518151040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829791.125138:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518150912:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829791.125153:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829791.125156:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829791.125158:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829791.125162:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2797 00000100:00080000:1.0:1787829791.125166:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829791.125169:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829791.125174:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829791.125186:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518151168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829791.125191:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518151040:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829791.125200:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829791.125204:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829791.125207:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829791.125211:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2797 00000100:00080000:1.0:1787829791.125215:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829791.125218:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829791.125221:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518151168:192.168.203.101@tcp:8:kworker.0 00000008:00100000:1.0:1787829791.125222:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829791.125239:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518151296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829791.125251:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518151296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829791.444769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829791.956734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829791.956801:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829792.148784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829792.148784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829792.468743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829792.980724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829792.980796:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829793.172766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829793.172802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829793.492733:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829794.004753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829794.004835:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829794.196788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829794.196788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:1.0:1787829794.196984:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787829794.197037:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829794.197056:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829794.197099:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829794.197107:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829794.197115:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829794.197123:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829794.197136:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787829794.197149:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829791/real 1787829794] req@ffff8b2e88076580 x1874672518151040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787829794.197150:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787829794.197179:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518151040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829794.197192:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829794.197196:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829794.197200:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518151040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829794.197207:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829791/real 1787829794] req@ffff8b2e88076d00 x1874672518150912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829794.197217:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518150912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829794.197223:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829794.197226:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829794.197228:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518150912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829794.197234:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829791/real 1787829794] req@ffff8b2e88077840 x1874672518150656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829794.197245:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518150656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829794.197254:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829794.197258:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829794.197261:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518150656:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829794.197270:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829791/real 1787829794] req@ffff8b2e880770c0 x1874672518150784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829794.197283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518150784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829794.197292:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829794.197296:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829794.197300:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518150784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829794.197308:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829791/real 1787829794] req@ffff8b2e88074b40 x1874672518150528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829794.197321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518150528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829794.197330:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829794.197333:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829794.197336:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518150528:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829794.197383:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829791/real 1787829794] req@ffff8b2e88074000 x1874672518151296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829794.197396:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518151296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829794.197406:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829794.197410:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829794.197413:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518151296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829794.197421:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829791/real 1787829794] req@ffff8b2e880761c0 x1874672518151168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829794.197434:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518151168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829816 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829794.197444:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829794.197448:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829794.197451:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518151168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829794.516736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829795.028717:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829795.028778:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829795.220735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829795.220755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829795.540767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829796.052796:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829796.052865:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829796.244789:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829796.244794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829796.244801:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787829796.244808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829796.244809:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829796.244823:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2802 00000100:00080000:3.0:1787829796.244834:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829796.244841:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829796.244874:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518151424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829796.244898:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829796.244901:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829796.244902:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829796.244905:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2802 00000100:00080000:3.0:1787829796.244907:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829796.244910:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829796.244919:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829796.244933:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518151552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829796.244938:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518151424:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829796.244953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829796.244957:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829796.244959:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829796.244964:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2802 00000100:00080000:3.0:1787829796.244968:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829796.244972:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829796.244978:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829796.244992:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518151680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829796.245002:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829796.245006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829796.245010:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829796.245011:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829796.245017:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2802 00000100:00080000:3.0:1787829796.245022:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829796.245027:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829796.245032:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829796.245045:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518151808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829796.245055:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829796.245057:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff8b2e8803b480 x1874672518151424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829796.245059:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829796.245063:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829796.245072:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2802 00000100:00080000:3.0:1787829796.245078:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829796.245083:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829796.245087:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518151424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829796.245090:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829796.245105:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829796.245108:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518151936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829796.245120:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829796.245120:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829796.245127:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829796.245129:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518151424:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829796.245130:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829796.245140:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2802 00000100:00080000:3.0:1787829796.245144:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829796.245148:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829796.245153:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829796.245163:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518151552:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829796.245168:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518152064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829796.245180:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829796.245184:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829796.245186:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829796.245189:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2802 00000100:00080000:3.0:1787829796.245194:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787829796.245199:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829796.245203:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829796.245210:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829796.245222:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff8b2e88039e00 x1874672518151552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829796.245225:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518152192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829796.245260:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518151552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.245278:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.245283:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829796.245287:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518151552:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829796.245294:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518151680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829796.245317:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829796.245329:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff8b2e88039a40 x1874672518151680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.245342:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518151680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.245351:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.245355:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829796.245358:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518151680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829796.245365:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518151808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829796.245386:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829796.245398:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff8b2e88039680 x1874672518151808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.245410:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518151808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.245419:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.245422:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829796.245426:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518151808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829796.245432:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518151936:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829796.245452:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829796.245464:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff8b2e880392c0 x1874672518151936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.245476:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518151936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.245485:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.245489:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829796.245492:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518151936:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829796.245509:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518152064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829796.245545:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829796.245558:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff8b2e880383c0 x1874672518152064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.245570:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518152064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.245578:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.245582:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829796.245585:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518152064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829796.245591:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518152192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829796.245616:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829796.245628:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829796/real 1787829796] req@ffff8b2e88038f00 x1874672518152192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829796.245673:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518152192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829821 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829796.245683:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829796.245687:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829796.245690:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518152192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829796.564747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829797.076730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829797.076869:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829797.268752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829797.268757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829797.588765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829798.100781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829798.100825:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829798.292758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829798.292762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829798.612712:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829799.124731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829799.124793:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829799.316736:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829799.316749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829799.636731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829800.148774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829800.148850:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829800.340783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829800.340804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829800.660705:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829801.172791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829801.172872:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829801.364765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829801.364770:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829801.364780:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:0.0:1787829801.364781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829801.364787:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829801.364801:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2807 00000100:00080000:3.0:1787829801.364817:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829801.364826:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829801.364872:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518152320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829801.364900:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829801.364903:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829801.364905:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829801.364908:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2807 00000100:00080000:3.0:1787829801.364911:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829801.364914:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829801.364923:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829801.364949:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518152448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829801.364950:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518152320:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829801.364975:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829801.364979:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829801.364981:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829801.364985:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2807 00000100:00080000:3.0:1787829801.364988:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829801.364991:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829801.364995:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829801.365006:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518152576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829801.365013:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829801.365016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829801.365018:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829801.365021:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2807 00000100:00080000:3.0:1787829801.365024:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829801.365027:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829801.365032:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829801.365034:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829801.365051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518152704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829801.365064:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829801.365069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829801.365071:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829801.365080:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2807 00000100:00080000:3.0:1787829801.365085:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829801.365092:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff8b2e8803a580 x1874672518152320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829801.365093:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829801.365108:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829801.365119:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518152320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829801.365129:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829801.365131:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518152832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829801.365135:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829801.365145:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518152320:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829801.365168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829801.365173:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829801.365176:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829801.365182:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518152448:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829801.365191:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2807 00000100:00080000:3.0:1787829801.365199:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829801.365206:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787829801.365213:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829801.365224:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff8b2e8803b840 x1874672518152448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829801.365224:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829801.365249:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518152960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829801.365250:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518152448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829801.365296:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829801.365304:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829801.365310:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518152448:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829801.365326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518152576:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787829801.365351:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829801.365354:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829801.365356:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829801.365358:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829801.365377:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2807 00000100:00100000:0.0:1787829801.365383:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff8b2e8803b0c0 x1874672518152576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829801.365383:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829801.365396:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829801.365411:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518152576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787829801.365411:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829801.365425:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829801.365434:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829801.365434:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518153088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829801.365446:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518152576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829801.365470:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518152704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829801.365487:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829801.365515:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff8b2e8803a1c0 x1874672518152704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829801.365524:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518152704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829801.365534:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829801.365537:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829801.365539:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518152704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829801.365543:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518152832:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829801.365558:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829801.365566:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff8b2e88038f00 x1874672518152832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829801.365573:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518152832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829801.365578:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829801.365581:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829801.365582:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518152832:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829801.365596:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518152960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829801.365607:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829801.365613:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff8b2e880383c0 x1874672518152960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829801.365621:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518152960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829801.365626:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829801.365627:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829801.365630:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518152960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829801.365634:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518153088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829801.365644:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829801.365672:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829801/real 1787829801] req@ffff8b2e880392c0 x1874672518153088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829801.365680:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518153088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829826 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829801.365684:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829801.365686:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829801.365688:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518153088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829801.684707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829802.196780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829802.196853:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829802.388762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829802.388763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829802.708715:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829803.220764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829803.220809:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829803.412715:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829803.412766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829803.732707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829804.244810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829804.244885:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829804.436757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829804.436783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829804.756789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829805.268772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829805.268864:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829805.460764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829805.460764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829805.780793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829806.292755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829806.292862:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829806.484755:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829806.484757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829806.484760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829806.484768:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829806.484773:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829806.484784:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2812 00000100:00080000:3.0:1787829806.484798:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829806.484806:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829806.484855:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518153216/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829806.484886:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829806.484890:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829806.484893:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829806.484898:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2812 00000100:00080000:3.0:1787829806.484903:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829806.484908:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829806.484920:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829806.484923:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518153216:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829806.484943:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518153344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829806.484954:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829806.484957:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829806.484959:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829806.484962:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2812 00000100:00080000:3.0:1787829806.484965:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829806.484968:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829806.484972:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829806.484983:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518153472/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829806.484989:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829806.484993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:0.0:1787829806.484995:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829806.484996:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829806.485002:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2812 00000100:00080000:3.0:1787829806.485005:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829806.485007:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829806.485010:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829806.485020:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518153600/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829806.485026:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829806.485029:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829806.485030:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829806.485034:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2812 00000100:00080000:3.0:1787829806.485036:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829806.485039:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829806.485042:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829806.485050:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff8b2e88039680 x1874672518153216/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829806.485054:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518153728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829806.485078:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829806.485081:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829806.485082:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829806.485086:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518153216/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829806.485089:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2812 00000100:00080000:3.0:1787829806.485094:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829806.485099:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829806.485105:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829806.485107:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829806.485113:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829806.485116:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518153856/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829806.485121:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518153216:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829806.485134:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829806.485137:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829806.485138:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829806.485141:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2812 00000100:00080000:3.0:1787829806.485143:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829806.485146:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829806.485149:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829806.485151:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518153344:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829806.485159:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518153984/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829806.485177:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829806.485195:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff8b2e88039a40 x1874672518153344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829806.485210:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518153344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829806.485220:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829806.485224:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829806.485228:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518153344:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829806.485234:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518153472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829806.485252:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829806.485259:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff8b2e88039e00 x1874672518153472/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829806.485268:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518153472/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829806.485275:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829806.485278:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829806.485282:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518153472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829806.485288:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518153600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829806.485308:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829806.485320:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff8b2e8803b480 x1874672518153600/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829806.485332:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518153600/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829806.485339:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829806.485341:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829806.485343:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518153600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829806.485347:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518153728:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829806.485362:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829806.485369:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff8b2e880392c0 x1874672518153728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829806.485376:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518153728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829806.485382:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829806.485384:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829806.485386:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518153728:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829806.485390:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518153856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829806.485399:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829806.485405:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff8b2e880383c0 x1874672518153856/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829806.485412:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518153856/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829806.485418:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829806.485420:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829806.485422:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518153856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829806.485434:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518153984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829806.485444:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829806.485450:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829806/real 1787829806] req@ffff8b2e88038f00 x1874672518153984/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829806.485479:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518153984/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829831 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829806.485486:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829806.485488:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829806.485490:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518153984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829806.804765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829807.316770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829807.316800:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829807.508778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829807.508783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829807.828746:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829808.340771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829808.340852:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829808.532744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829808.532751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829808.852768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829809.364815:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829809.364903:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829809.556791:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829809.556987:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829809.876756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829810.388748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829810.388823:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829810.580733:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829810.580738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829810.900769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829811.412760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829811.412818:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829811.604747:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829811.604752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829811.604753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829811.604762:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829811.604767:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829811.604778:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2818 00000100:00080000:3.0:1787829811.604790:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829811.604799:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829811.604837:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518154112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829811.604859:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.604862:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829811.604864:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829811.604867:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2818 00000100:00080000:3.0:1787829811.604869:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829811.604872:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829811.604881:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829811.604890:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518154240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829811.604899:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518154112:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829811.604900:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.604907:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829811.604911:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829811.604915:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2818 00000100:00080000:3.0:1787829811.604917:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829811.604919:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829811.604923:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829811.604932:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518154368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829811.604939:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.604942:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829811.604943:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829811.604946:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2818 00000100:00080000:3.0:1787829811.604948:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829811.604950:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829811.604953:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829811.604959:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518154496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829811.604965:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.604967:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829811.604969:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829811.604973:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829811.604974:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2818 00000100:00080000:3.0:1787829811.604979:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829811.604981:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829811.604985:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829811.605004:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518154624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829811.605015:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff8b2e8803a1c0 x1874672518154112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829811.605016:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.605028:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829811.605033:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829811.605038:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2818 00000100:00100000:0.0:1787829811.605040:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518154112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829811.605043:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829811.605053:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829811.605055:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829811.605059:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829811.605063:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829811.605069:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518154112:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829811.605076:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518154752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829811.605086:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829811.605089:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829811.605092:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829811.605095:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518154240:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829811.605099:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2818 00000100:00080000:3.0:1787829811.605107:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829811.605111:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829811.605116:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829811.605127:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829811.605132:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518154880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829811.605144:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff8b2e8803b0c0 x1874672518154240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829811.605169:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518154240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829811.605179:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829811.605182:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829811.605184:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518154240:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829811.605189:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518154368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829811.605207:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829811.605215:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff8b2e8803b840 x1874672518154368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829811.605230:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518154368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829811.605236:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829811.605238:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829811.605240:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518154368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829811.605244:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518154496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829811.605256:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829811.605262:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff8b2e8803a580 x1874672518154496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829811.605270:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518154496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829811.605281:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829811.605283:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829811.605285:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518154496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829811.605289:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518154624:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829811.605300:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829811.605306:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff8b2e88038f00 x1874672518154624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829811.605314:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518154624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829811.605319:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829811.605321:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829811.605323:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518154624:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829811.605326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518154752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829811.605336:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829811.605341:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff8b2e880383c0 x1874672518154752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829811.605349:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518154752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829811.605354:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829811.605356:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829811.605357:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518154752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829811.605370:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518154880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829811.605380:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829811.605385:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829811/real 1787829811] req@ffff8b2e880392c0 x1874672518154880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829811.605393:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518154880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829836 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829811.605400:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829811.605403:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829811.605407:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518154880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829811.924748:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829812.436721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829812.436783:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829812.628727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829812.628758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829812.948707:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829813.460776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829813.460829:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829813.652784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829813.652784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829813.972711:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829814.484735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829814.484839:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829814.676775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829814.676793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829814.996794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829815.508777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829815.508860:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829815.700829:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829815.700830:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829816.020752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829816.532798:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829816.532915:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829816.724755:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829816.724759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829816.724759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829816.724770:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829816.724777:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.724789:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2823 00000100:00080000:3.0:1787829816.724804:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.724816:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829816.724870:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518155008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829816.724901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.724904:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829816.724906:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.724911:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2823 00000100:00080000:3.0:1787829816.724916:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.724921:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829816.724935:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829816.724946:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518155008:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829816.724954:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518155136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829816.724967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.724970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829816.724972:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.724975:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2823 00000100:00080000:3.0:1787829816.724979:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.724981:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829816.724986:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829816.724998:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518155264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829816.725005:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.725008:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829816.725009:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.725011:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2823 00000100:00080000:3.0:1787829816.725016:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.725021:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829816.725025:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829816.725039:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518155392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829816.725050:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829816.725057:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.725062:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829816.725065:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.725071:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2823 00000100:00080000:3.0:1787829816.725075:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.725079:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829816.725083:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829816.725094:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518155520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829816.725101:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829816.725104:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829816.725105:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829816.725110:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2823 00000100:00100000:0.0:1787829816.725112:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff8b2e8803b480 x1874672518155008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829816.725113:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829816.725119:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829816.725125:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829816.725142:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518155648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829816.725149:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518155008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829816.725188:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829816.725189:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829816.725191:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829816.725195:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829816.725200:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829816.725201:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2823 00000100:00080000:3.0:1787829816.725208:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829816.725211:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518155008:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829816.725217:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829816.725226:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829816.725241:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518155776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829816.725257:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518155136:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829816.725288:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829816.725304:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff8b2e88039e00 x1874672518155136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829816.725319:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518155136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829816.725330:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829816.725333:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829816.725337:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518155136:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829816.725344:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518155264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829816.725367:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829816.725380:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff8b2e88039a40 x1874672518155264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829816.725412:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518155264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829816.725428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829816.725432:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829816.725435:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518155264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829816.725441:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518155392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829816.725463:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829816.725477:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff8b2e88039680 x1874672518155392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829816.725499:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518155392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829816.725510:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829816.725514:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829816.725518:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518155392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829816.725525:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518155520:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829816.725557:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829816.725574:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff8b2e880392c0 x1874672518155520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829816.725588:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518155520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829816.725598:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829816.725601:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829816.725606:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518155520:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829816.725612:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518155648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829816.725635:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829816.725671:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff8b2e880383c0 x1874672518155648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829816.725686:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518155648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829816.725695:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829816.725698:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829816.725702:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518155648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829816.725723:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518155776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829816.725746:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829816.725759:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829816/real 1787829816] req@ffff8b2e88038f00 x1874672518155776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829816.725773:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518155776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829841 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829816.725782:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829816.725786:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829816.725789:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518155776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829817.044756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829817.556774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829817.556851:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829817.748767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829817.748768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829818.068736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829818.580709:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829818.580867:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829818.772782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829818.772782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829819.092789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829819.604761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829819.604834:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829819.796763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829819.796770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829820.116743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829820.628778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829820.628904:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829820.820769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829820.820874:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829821.140785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829821.652809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829821.652876:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829821.844769:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829821.844771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829821.844771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829821.844781:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829821.844787:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829821.844798:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2828 00000100:00080000:3.0:1787829821.844809:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829821.844818:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829821.844857:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518155904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829821.844884:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829821.844888:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829821.844890:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829821.844894:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2828 00000100:00080000:3.0:1787829821.844899:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829821.844905:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829821.844916:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829821.844930:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518155904:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829821.844930:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518156032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829821.844946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829821.844948:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829821.844949:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829821.844952:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2828 00000100:00080000:3.0:1787829821.844955:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829821.844958:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829821.844961:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829821.844969:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518156160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829821.844976:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829821.844978:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829821.844980:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829821.844982:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2828 00000100:00080000:3.0:1787829821.844984:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829821.844986:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829821.844988:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829821.845003:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518156288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829821.845006:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829821.845014:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829821.845017:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829821.845019:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829821.845023:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2828 00000100:00080000:3.0:1787829821.845026:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829821.845029:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829821.845032:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829821.845045:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518156416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829821.845052:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829821.845055:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829821.845056:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829821.845061:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2828 00000100:00100000:0.0:1787829821.845063:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff8b2e8803a580 x1874672518155904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829821.845063:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829821.845068:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829821.845073:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829821.845083:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518156544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829821.845093:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518155904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829821.845096:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829821.845104:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829821.845106:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829821.845113:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2828 00000100:00080000:3.0:1787829821.845118:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829821.845120:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829821.845121:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829821.845123:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829821.845130:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829821.845134:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518156672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829821.845140:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518155904:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829821.845169:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518156032:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829821.845209:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829821.845224:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff8b2e8803b840 x1874672518156032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829821.845239:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518156032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829821.845249:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829821.845253:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829821.845257:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518156032:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829821.845264:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518156160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829821.845290:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829821.845303:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff8b2e8803b0c0 x1874672518156160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829821.845317:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518156160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829821.845327:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829821.845330:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829821.845334:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518156160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829821.845341:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518156288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829821.845380:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829821.845392:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff8b2e8803a1c0 x1874672518156288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829821.845405:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518156288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829821.845415:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829821.845418:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829821.845422:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518156288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829821.845428:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518156416:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829821.845451:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829821.845463:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff8b2e88038f00 x1874672518156416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829821.845484:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518156416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829821.845503:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829821.845506:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829821.845510:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518156416:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829821.845516:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518156544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829821.845538:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829821.845551:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff8b2e880383c0 x1874672518156544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829821.845564:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518156544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829821.845573:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829821.845577:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829821.845580:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518156544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829821.845586:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518156672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829821.845607:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829821.845620:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829821/real 1787829821] req@ffff8b2e880392c0 x1874672518156672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829821.845632:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518156672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829846 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829821.845669:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829821.845673:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829821.845677:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518156672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829822.164777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829822.676788:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829822.676864:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829822.868777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829822.868778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829823.188760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829823.700774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829823.700839:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829823.892747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829823.892760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829824.212766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829824.724752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829824.724835:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829824.916759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829824.916759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829825.236744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829825.748756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829825.748821:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829825.940718:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829825.940724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829826.260764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829826.772773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829826.772829:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829826.964752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829826.964769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829826.964769:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.964781:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829826.964786:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.964798:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2833 00000100:00080000:3.0:1787829826.964820:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.964833:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829826.964873:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518156800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829826.964897:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.964902:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829826.964904:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.964909:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2833 00000100:00080000:3.0:1787829826.964913:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.964918:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829826.964929:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518156800:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787829826.964929:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829826.964947:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518156928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829826.964958:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.964962:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829826.964964:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.964968:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2833 00000100:00080000:3.0:1787829826.964973:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.964978:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829826.964983:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829826.964994:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518157056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829826.965000:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829826.965005:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829826.965005:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829826.965008:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.965012:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2833 00000100:00080000:3.0:1787829826.965015:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.965017:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829826.965020:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829826.965028:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518157184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829826.965035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.965039:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829826.965041:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.965045:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2833 00000100:00100000:0.0:1787829826.965055:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff8b2e88039680 x1874672518156800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829826.965058:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.965067:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829826.965072:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829826.965076:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518156800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829826.965085:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829826.965089:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518157312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829826.965091:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829826.965098:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518156800:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829826.965114:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.965119:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829826.965121:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518156928:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829826.965121:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.965128:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2833 00000100:00080000:3.0:1787829826.965132:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.965136:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829826.965140:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829826.965146:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829826.965152:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518157440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829826.965159:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff8b2e88039a40 x1874672518156928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829826.965161:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829826.965168:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829826.965171:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829826.965179:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2833 00000100:00100000:0.0:1787829826.965182:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518156928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829826.965182:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829826.965186:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829826.965191:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829826.965196:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787829826.965201:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518157568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829826.965202:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829826.965209:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518156928:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829826.965217:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518157056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829826.965232:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829826.965238:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff8b2e88039e00 x1874672518157056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829826.965244:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518157056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829826.965249:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829826.965251:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829826.965253:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518157056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829826.965257:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518157184:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829826.965266:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829826.965271:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff8b2e8803b480 x1874672518157184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829826.965277:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518157184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829826.965282:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829826.965283:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829826.965286:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518157184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829826.965289:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518157312:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829826.965302:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829826.965311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff8b2e880392c0 x1874672518157312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829826.965333:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518157312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829826.965342:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829826.965344:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829826.965347:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518157312:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829826.965364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518157440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829826.965384:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829826.965395:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff8b2e880383c0 x1874672518157440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829826.965406:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518157440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829826.965414:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829826.965417:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829826.965420:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518157440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829826.965426:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518157568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829826.965444:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829826.965465:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829826/real 1787829826] req@ffff8b2e88038f00 x1874672518157568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829826.965476:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518157568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829851 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829826.965485:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829826.965487:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829826.965490:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518157568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829827.284759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829827.796782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829827.796853:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829827.988720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829827.988720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829828.308772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829828.820775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829828.820795:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829829.012792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829829.012792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829829.332783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829829.844774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829829.844842:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829830.036760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829830.036761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829830.356756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829830.868771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829830.868827:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829831.060767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829831.060768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829831.380794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829831.892767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829831.892854:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829832.084796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829832.084797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829832.084799:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829832.084813:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829832.084820:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829832.084831:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2838 00000100:00080000:3.0:1787829832.084843:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829832.084851:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829832.084893:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518157696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829832.084920:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829832.084924:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829832.084926:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829832.084930:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2838 00000100:00080000:3.0:1787829832.084932:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829832.084936:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829832.084946:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829832.084959:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518157824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829832.084968:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518157696:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829832.084985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829832.084990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829832.084992:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829832.084997:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2838 00000100:00080000:3.0:1787829832.085000:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829832.085003:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829832.085009:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829832.085020:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518157952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829832.085030:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829832.085033:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829832.085036:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829832.085038:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829832.085050:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2838 00000100:00080000:3.0:1787829832.085058:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829832.085064:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829832.085070:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829832.085084:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829832/real 1787829832] req@ffff8b2e8803a1c0 x1874672518157696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829832.085087:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518158080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829832.085124:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829832.085126:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518157696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829832.085142:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829832.085149:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829832.085150:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829832.085164:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829832.085164:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2838 00000100:00100000:0.0:1787829832.085173:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518157696:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829832.085173:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829832.085187:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829832.085193:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829832.085201:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518157824:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829832.085209:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518158208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829832.085224:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829832.085224:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829832.085235:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829832.085237:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829832.085242:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829832/real 1787829832] req@ffff8b2e8803b0c0 x1874672518157824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829832.085246:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2838 00000100:00080000:3.0:1787829832.085255:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829832.085262:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518157824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829832.085263:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829832.085289:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829832.085294:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829832.085298:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829832.085302:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518157824:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829832.085307:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518158336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829832.085308:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518157952:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787829832.085338:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829832.085339:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829832.085349:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829832.085352:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829832.085359:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829832/real 1787829832] req@ffff8b2e8803b840 x1874672518157952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829832.085362:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2838 00000100:00080000:3.0:1787829832.085368:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829832.085377:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829832.085388:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829832.085395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518157952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829832.085403:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518158464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829832.085428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829832.085439:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829832.085447:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518157952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829832.085455:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518158080:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829832.085473:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829832.085480:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829832/real 1787829832] req@ffff8b2e8803a580 x1874672518158080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829832.085487:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518158080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829832.085491:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829832.085493:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829832.085495:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518158080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829832.085507:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518158208:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829832.085523:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829832.085528:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829832/real 1787829832] req@ffff8b2e8803a1c0 x1874672518158208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829832.085534:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518158208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829832.085539:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829832.085541:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829832.085543:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518158208:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829832.085546:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518158336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829832.085554:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829832.085559:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829832/real 1787829832] req@ffff8b2e88038f00 x1874672518158336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829832.085565:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518158336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829832.085570:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829832.085571:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829832.085574:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518158336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829832.085577:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518158464:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829832.085585:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829832.085590:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829832/real 1787829832] req@ffff8b2e880383c0 x1874672518158464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829832.085595:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518158464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829857 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829832.085600:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829832.085601:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829832.085603:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518158464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829832.404757:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829832.916761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829832.916822:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829833.108776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829833.108776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829833.428756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829833.940735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829833.940799:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829834.132765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829834.132765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829834.452750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829834.964731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829834.964816:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829835.156752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829835.156775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829835.476715:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829835.988721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829835.988796:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829836.180742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829836.180749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829836.500785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829837.012758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829837.012833:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829837.204750:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829837.204758:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829837.204763:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829837.204765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829837.204767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829837.204776:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2843 00000100:00080000:3.0:1787829837.204790:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829837.204801:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829837.204855:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518158592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829837.204883:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829837.204887:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829837.204889:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829837.204895:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2843 00000100:00080000:3.0:1787829837.204898:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829837.204902:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829837.204908:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518158592:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787829837.204912:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829837.204931:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518158720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829837.204942:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829837.204945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829837.204948:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829837.204952:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2843 00000100:00080000:3.0:1787829837.204956:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829837.204960:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829837.204965:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829837.204975:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518158848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829837.204980:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829837.204990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829837.204994:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829837.204996:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829837.205001:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2843 00000100:00080000:3.0:1787829837.205005:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829837.205009:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829837.205015:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829837.205021:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829837/real 1787829837] req@ffff8b2e880392c0 x1874672518158592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829837.205026:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518158976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829837.205038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829837.205041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:0.0:1787829837.205043:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518158592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829837.205044:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829837.205051:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2843 00000100:00080000:3.0:1787829837.205056:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829837.205061:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787829837.205064:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787829837.205065:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787829837.205073:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829837.205075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518159104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829837.205083:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518158592:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829837.205084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829837.205090:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829837.205092:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829837.205095:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2843 00000100:00080000:3.0:1787829837.205097:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829837.205099:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829837.205102:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829837.205108:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518159232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829837.205112:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518158720:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829837.205118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829837.205120:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829837.205122:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829837.205125:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2843 00000100:00080000:3.0:1787829837.205128:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829837.205130:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829837.205133:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829837.205137:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829837.205143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518159360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829837.205149:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829837/real 1787829837] req@ffff8b2e8803b480 x1874672518158720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829837.205159:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518158720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829837.205166:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829837.205168:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829837.205171:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518158720:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829837.205176:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518158848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829837.205188:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829837.205193:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829837/real 1787829837] req@ffff8b2e88039e00 x1874672518158848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829837.205200:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518158848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829837.205205:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829837.205206:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829837.205208:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518158848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829837.205211:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518158976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829837.205220:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829837.205227:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829837/real 1787829837] req@ffff8b2e88039a40 x1874672518158976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829837.205238:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518158976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829837.205256:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829837.205260:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829837.205263:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518158976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829837.205269:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518159104:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829837.205287:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829837.205294:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829837/real 1787829837] req@ffff8b2e88039680 x1874672518159104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829837.205301:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518159104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829837.205307:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829837.205309:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829837.205310:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518159104:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829837.205321:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518159232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829837.205330:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829837.205336:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829837/real 1787829837] req@ffff8b2e880392c0 x1874672518159232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829837.205342:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518159232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829837.205348:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829837.205350:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829837.205352:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518159232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829837.205355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518159360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829837.205362:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829837.205367:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829837/real 1787829837] req@ffff8b2e880383c0 x1874672518159360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829837.205373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518159360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829862 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829837.205378:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829837.205379:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829837.205381:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518159360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829837.524719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829838.036780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829838.036856:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829838.228746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829838.228751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829838.548743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829839.060758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829839.060825:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829839.252741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829839.252748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829839.572717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829840.084749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829840.084827:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829840.276740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829840.276742:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829840.596769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829841.108755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829841.108814:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829841.300770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829841.300781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829841.620747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829842.132756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829842.132915:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829842.324753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829842.324762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829842.324765:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829842.324774:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829842.324779:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829842.324794:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2848 00000100:00080000:1.0:1787829842.324809:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829842.324832:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829842.324878:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518159488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829842.324906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829842.324909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829842.324911:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829842.324915:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2848 00000100:00080000:1.0:1787829842.324919:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829842.324924:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829842.324935:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829842.324947:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518159488:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829842.324949:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518159616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829842.324963:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829842.324966:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829842.324968:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829842.324973:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2848 00000100:00080000:1.0:1787829842.324976:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829842.324980:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829842.324985:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829842.324996:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518159744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829842.325014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829842.325017:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829842.325019:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829842.325026:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2848 00000800:00000100:0.0:1787829842.325027:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829842.325032:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829842.325037:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829842.325043:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829842.325055:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518159872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829842.325064:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829842.325067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829842.325069:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829842.325074:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2848 00000100:00080000:1.0:1787829842.325078:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829842.325079:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff8b2e88075e00 x1874672518159488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829842.325084:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829842.325092:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829842.325106:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518159488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829842.325107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518160000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829842.325133:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829842.325134:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829842.325139:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829842.325141:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787829842.325145:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518159488:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829842.325145:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829842.325154:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2848 00000100:00080000:1.0:1787829842.325158:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829842.325162:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829842.325166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829842.325171:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518159616:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829842.325179:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518160128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829842.325189:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787829842.325193:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829842.325196:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829842.325200:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829842.325205:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2848 00000100:00100000:0.0:1787829842.325223:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff8b2e88074780 x1874672518159616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829842.325223:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829842.325231:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829842.325239:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829842.325247:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518159616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829842.325254:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518160256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829842.325262:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829842.325269:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829842.325273:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518159616:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829842.325278:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518159744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829842.325296:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829842.325304:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff8b2e88077c00 x1874672518159744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829842.325312:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518159744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829842.325318:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829842.325320:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829842.325322:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518159744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829842.325326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518159872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829842.325336:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829842.325341:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff8b2e88075680 x1874672518159872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829842.325348:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518159872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829842.325353:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829842.325354:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829842.325356:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518159872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829842.325359:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518160000:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829842.325369:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829842.325374:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff8b2e88075a40 x1874672518160000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829842.325380:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518160000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829842.325386:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829842.325388:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829842.325389:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518160000:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829842.325400:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518160128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829842.325410:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829842.325415:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff8b2e88077480 x1874672518160128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829842.325423:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518160128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829842.325428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829842.325430:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829842.325432:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518160128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829842.325435:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518160256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829842.325442:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829842.325447:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829842/real 1787829842] req@ffff8b2e88076940 x1874672518160256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829842.325453:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518160256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829867 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829842.325466:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829842.325468:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829842.325470:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518160256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829842.644747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829843.156862:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829843.156905:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829843.348752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829843.348754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829843.668753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829844.180832:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829844.180871:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787829844.372735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829844.372735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829844.692747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829845.204759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829845.204833:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829845.396734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829845.396735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829845.716753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829846.228763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829846.228826:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829846.420745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829846.420746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829846.740703:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829847.252749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829847.252821:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829847.444726:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829847.444766:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829847.444767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829847.444776:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829847.444781:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829847.444798:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2853 00000100:00080000:3.0:1787829847.444811:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829847.444821:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829847.444868:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518160384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829847.444892:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829847.444894:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829847.444896:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829847.444899:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2853 00000100:00080000:3.0:1787829847.444902:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829847.444905:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829847.444914:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829847.444925:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518160512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829847.444933:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829847.444936:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829847.444937:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829847.444940:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2853 00000100:00080000:3.0:1787829847.444943:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829847.444947:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518160384:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829847.444948:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829847.444957:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829847.444970:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518160640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829847.444975:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829847.444978:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829847.444979:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829847.444982:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2853 00000100:00080000:3.0:1787829847.444987:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829847.444989:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829847.444992:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829847.444998:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518160768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829847.445003:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829847.445006:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829847.445006:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829847.445010:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2853 00000100:00080000:3.0:1787829847.445015:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829847.445018:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829847.445021:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829847.445028:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518160896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829847.445034:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829847.445037:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829847.445038:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829847.445047:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829847.445048:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2853 00000100:00080000:3.0:1787829847.445055:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829847.445058:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829847.445061:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829847.445070:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518161024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829847.445077:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829847.445079:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829847.445079:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829847.445083:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2853 00000100:00080000:3.0:1787829847.445085:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829847.445088:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829847.445092:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829847.445101:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518161152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829847.445112:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff8b2e88038f00 x1874672518160384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829847.445135:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518160384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829847.445147:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829847.445152:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829847.445156:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518160384:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829847.445200:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518160512:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829847.445228:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829847.445242:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff8b2e8803a1c0 x1874672518160512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829847.445257:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518160512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829847.445267:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829847.445270:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829847.445274:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518160512:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829847.445281:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518160640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829847.445303:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829847.445315:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff8b2e8803a580 x1874672518160640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829847.445328:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518160640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829847.445343:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829847.445347:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829847.445350:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518160640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829847.445357:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518160768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829847.445378:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829847.445390:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff8b2e8803b840 x1874672518160768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829847.445402:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518160768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829847.445411:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829847.445415:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829847.445418:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518160768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829847.445424:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518160896:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829847.445444:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829847.445456:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff8b2e8803b0c0 x1874672518160896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829847.445468:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518160896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829847.445476:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829847.445480:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829847.445483:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518160896:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829847.445490:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518161024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829847.445510:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829847.445522:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff8b2e880383c0 x1874672518161024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829847.445535:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518161024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829847.445544:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829847.445547:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829847.445551:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518161024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829847.445557:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518161152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829847.445577:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829847.445590:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829847/real 1787829847] req@ffff8b2e880392c0 x1874672518161152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829847.445603:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518161152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829872 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829847.445612:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829847.445615:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829847.445619:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518161152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829847.764782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829848.276746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829848.276851:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829848.468797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829848.468799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829848.788772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829849.300793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829849.300867:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829849.492729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829849.492737:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829849.812760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829850.324774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829850.324861:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829850.516756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829850.516864:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829850.836763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829851.348784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829851.348858:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829851.540748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829851.540762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829851.860781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829852.372804:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829852.372864:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829852.564751:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829852.564764:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787829852.564767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829852.564768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787829852.564769:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.564788:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2858 00000100:00080000:1.0:1787829852.564803:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.564815:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829852.564859:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518161280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829852.564885:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829852.564888:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829852.564889:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.564892:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2858 00000100:00080000:1.0:1787829852.564895:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.564899:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829852.564910:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829852.564922:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518161408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829852.564934:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518161280:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829852.564937:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829852.564945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829852.564948:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.564952:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2858 00000100:00080000:1.0:1787829852.564955:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.564957:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829852.564961:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829852.564971:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518161536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829852.564978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829852.564980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829852.564981:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.564983:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2858 00000100:00080000:1.0:1787829852.564985:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.564987:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829852.564990:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829852.564996:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518161664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829852.565001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829852.565005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829852.565006:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.565011:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2858 00000100:00080000:1.0:1787829852.565025:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.565028:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829852.565034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829852.565041:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829852.565066:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518161792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829852.565079:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829852.565082:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829852.565084:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829852.565089:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2858 00000100:00080000:1.0:1787829852.565095:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829852.565096:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff8b2e88074f00 x1874672518161280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829852.565104:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829852.565118:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829852.565134:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518161280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829852.565152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518161920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829852.565185:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829852.565187:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829852.565194:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829852.565195:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787829852.565201:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787829852.565206:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518161280:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829852.565213:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2858 00000100:00080000:1.0:1787829852.565220:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829852.565226:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829852.565232:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829852.565242:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518161408:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829852.565251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518162048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787829852.565282:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829852.565304:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff8b2e88076940 x1874672518161408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829852.565318:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518161408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829852.565328:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829852.565331:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829852.565335:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518161408:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829852.565343:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518161536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829852.565368:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829852.565380:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff8b2e88077480 x1874672518161536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829852.565393:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518161536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829852.565401:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829852.565404:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829852.565408:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518161536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829852.565415:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518161664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829852.565434:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829852.565444:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff8b2e88075a40 x1874672518161664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829852.565456:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518161664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829852.565464:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829852.565468:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829852.565471:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518161664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829852.565478:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518161792:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829852.565497:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829852.565508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff8b2e88075680 x1874672518161792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829852.565527:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518161792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829852.565536:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829852.565538:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829852.565541:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518161792:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829852.565548:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518161920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829852.565568:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829852.565580:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff8b2e88077c00 x1874672518161920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829852.565592:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518161920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829852.565600:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829852.565603:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829852.565606:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518161920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829852.565626:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518162048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829852.565676:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829852.565690:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829852/real 1787829852] req@ffff8b2e88074780 x1874672518162048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829852.565702:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518162048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829877 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829852.565716:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829852.565719:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829852.565722:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518162048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829852.884788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829853.396738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829853.396788:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829853.588760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829853.588761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829853.908769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829854.420783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829854.420870:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787829854.612811:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829854.612811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829854.932796:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829855.444777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829855.444839:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829855.636766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829855.636772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829855.956766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829856.468798:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829856.468867:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829856.660740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787829856.660791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829856.980726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829857.492736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829857.492832:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787829857.684810:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829857.684823:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787829857.684826:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829857.684831:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829857.684833:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829857.684846:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2864 00000100:00080000:3.0:1787829857.684862:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829857.684870:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829857.684912:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518162176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829857.684939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829857.684942:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829857.684944:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829857.684948:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2864 00000100:00080000:3.0:1787829857.684950:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829857.684953:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829857.684963:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829857.684973:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518162304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829857.684982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829857.684987:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518162176:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829857.684987:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829857.684991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829857.684998:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2864 00000100:00080000:3.0:1787829857.685001:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829857.685003:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829857.685006:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829857.685016:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518162432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829857.685023:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829857.685026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829857.685027:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829857.685030:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2864 00000100:00080000:3.0:1787829857.685033:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829857.685036:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829857.685040:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829857.685046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518162560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829857.685053:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829857.685056:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829857.685057:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829857.685061:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2864 00000100:00080000:3.0:1787829857.685063:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829857.685066:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829857.685068:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829857.685076:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518162688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829857.685083:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829857.685085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829857.685087:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829857.685091:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2864 00000100:00080000:3.0:1787829857.685093:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829857.685096:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829857.685113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829857.685116:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518162304:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829857.685128:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518162816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829857.685137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829857.685140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829857.685142:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829857.685146:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2864 00000100:00080000:3.0:1787829857.685149:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829857.685153:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518162432:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787829857.685155:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829857.685166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829857.685178:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518162944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829857.685190:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518162560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829857.685221:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518162688:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829857.685248:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518162816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829857.685277:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518162944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829858.004771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829858.516780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829858.516842:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829858.708778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829858.708875:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829859.028725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829859.540775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829859.540851:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829859.732720:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829859.732806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829860.052772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829860.564790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829860.564894:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829860.756793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829860.757044:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:1.0:1787829860.757049:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787829860.757110:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787829860.757128:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829860.757166:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829860.757176:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829860.757186:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829860.757196:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787829860.757208:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787829860.757213:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829860] req@ffff8b2e88039a40 x1874672518162304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787829860.757231:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787829860.757253:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518162304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829860.757271:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829860.757277:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829860.757282:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518162304:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829860.757291:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829860] req@ffff8b2e88039e00 x1874672518162432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829860.757306:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518162432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829860.757314:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829860.757317:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829860.757319:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518162432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829860.757325:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829860] req@ffff8b2e8803b480 x1874672518162560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829860.757337:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518162560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829860.757346:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829860.757349:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829860.757353:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518162560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829860.757359:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829860] req@ffff8b2e880392c0 x1874672518162688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829860.757370:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518162688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829860.757378:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829860.757381:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829860.757386:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518162688:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829860.757392:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829860] req@ffff8b2e88039680 x1874672518162176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829860.757404:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518162176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829860.757413:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829860.757416:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829860.757420:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518162176:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829860.757449:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829860] req@ffff8b2e880383c0 x1874672518162816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829860.757463:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518162816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829860.757471:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829860.757475:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829860.757478:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518162816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829860.757484:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829857/real 1787829860] req@ffff8b2e8803b0c0 x1874672518162944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829860.757495:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518162944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829882 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829860.757503:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829860.757507:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829860.757511:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518162944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829861.076765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829861.588773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829861.588801:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829861.780703:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829861.780786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829862.100732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829862.612813:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829862.612865:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:3.0:1787829862.804750:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.804761:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829862.804766:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.804777:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2869 00000100:00080000:3.0:1787829862.804790:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.804797:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829862.804812:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829862.804836:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518163072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829862.804838:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787829862.804868:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.804872:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829862.804874:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.804878:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2869 00000100:00080000:3.0:1787829862.804882:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.804886:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829862.804897:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829862.804905:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518163072:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829862.804910:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518163200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829862.804919:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.804922:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829862.804924:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.804927:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2869 00000100:00080000:3.0:1787829862.804930:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.804934:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829862.804938:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829862.804946:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518163328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829862.804954:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.804956:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829862.804958:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787829862.804962:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787829862.804962:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2869 00000100:00080000:3.0:1787829862.804965:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.804968:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829862.804973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829862.804981:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518163456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829862.804988:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.804990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829862.804992:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.804996:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2869 00000100:00080000:3.0:1787829862.804999:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787829862.805003:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff8b2e8803b840 x1874672518163072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829862.805003:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829862.805007:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829862.805017:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518163584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829862.805019:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518163072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829862.805027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829862.805030:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829862.805030:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829862.805031:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829862.805034:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829862.805035:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2869 00000100:00100000:0.0:1787829862.805038:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518163072:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787829862.805038:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.805041:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829862.805046:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829862.805054:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518163712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829862.805059:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518163200:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829862.805080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829862.805084:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829862.805086:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829862.805090:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2869 00000100:00080000:3.0:1787829862.805093:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829862.805097:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787829862.805100:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:3.0:1787829862.805101:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829862.805113:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff8b2e8803a580 x1874672518163200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829862.805113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518163840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829862.805126:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518163200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829862.805135:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829862.805138:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829862.805142:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518163200:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829862.805148:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518163328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829862.805169:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829862.805181:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff8b2e8803a1c0 x1874672518163328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829862.805194:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518163328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829862.805202:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829862.805205:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829862.805209:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518163328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829862.805215:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518163456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829862.805235:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829862.805247:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff8b2e88038f00 x1874672518163456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829862.805258:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518163456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829862.805267:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829862.805270:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829862.805274:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518163456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829862.805279:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518163584:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829862.805300:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829862.805311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff8b2e8803b0c0 x1874672518163584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829862.805321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518163584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829862.805330:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829862.805332:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829862.805335:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518163584:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829862.805354:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518163712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829862.805374:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829862.805384:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff8b2e880383c0 x1874672518163712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829862.805395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518163712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829862.805403:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829862.805406:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829862.805409:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518163712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829862.805415:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518163840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829862.805433:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829862.805443:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829862/real 1787829862] req@ffff8b2e88039680 x1874672518163840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829862.805453:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518163840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829887 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829862.805461:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829862.805464:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829862.805467:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518163840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829863.124763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829863.636761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829863.636809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829863.828771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829863.828866:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829864.148732:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829864.660759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829864.660838:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829864.852764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829864.852772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829865.172706:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829865.684741:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829865.684790:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829865.876763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829865.876763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829866.196718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829866.708726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829866.708815:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829866.900763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829866.900763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829867.220724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829867.732771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829867.732798:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829867.924814:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829867.924815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787829867.924818:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.924832:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787829867.924837:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.924851:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2874 00000100:00080000:3.0:1787829867.924864:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.924872:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829867.924920:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518163968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829867.924947:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.924950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829867.924952:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.924956:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2874 00000100:00080000:3.0:1787829867.924960:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.924964:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829867.924974:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829867.924981:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518163968:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829867.924989:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518164096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829867.924998:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.925001:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829867.925002:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.925005:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2874 00000100:00080000:3.0:1787829867.925008:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.925011:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829867.925015:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787829867.925026:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518164224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829867.925047:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.925051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829867.925053:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.925056:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2874 00000100:00080000:3.0:1787829867.925059:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.925062:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787829867.925067:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:3.0:1787829867.925067:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829867.925081:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518164352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829867.925089:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787829867.925092:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787829867.925093:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.925096:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2874 00000100:00080000:3.0:1787829867.925098:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.925100:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829867.925104:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829867.925115:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff8b2e880392c0 x1874672518163968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829867.925115:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518164480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829867.925151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829867.925157:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518163968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829867.925158:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787829867.925162:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.925173:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2874 00000100:00080000:0.0:1787829867.925177:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829867.925179:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829867.925189:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787829867.925190:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829867.925200:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829867.925201:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518163968:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829867.925220:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518164608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829867.925234:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787829867.925237:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518164096:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787829867.925238:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829867.925241:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829867.925248:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2874 00000100:00080000:3.0:1787829867.925251:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829867.925254:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787829867.925259:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787829867.925273:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829867.925274:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518164736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787829867.925296:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff8b2e8803b480 x1874672518164096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829867.925313:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518164096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829867.925324:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829867.925328:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829867.925331:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518164096:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829867.925338:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518164224:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829867.925362:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829867.925375:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff8b2e88039e00 x1874672518164224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829867.925389:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518164224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829867.925398:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829867.925400:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829867.925403:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518164224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829867.925407:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518164352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829867.925423:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829867.925430:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff8b2e88039a40 x1874672518164352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829867.925438:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518164352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829867.925443:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829867.925446:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829867.925448:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518164352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829867.925451:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518164480:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787829867.925461:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829867.925466:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff8b2e88039680 x1874672518164480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829867.925473:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518164480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829867.925480:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829867.925482:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829867.925484:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518164480:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829867.925496:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518164608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829867.925506:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829867.925511:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff8b2e880383c0 x1874672518164608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829867.925519:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518164608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829867.925524:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829867.925526:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829867.925528:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518164608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829867.925532:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518164736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787829867.925541:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829867.925555:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829867/real 1787829867] req@ffff8b2e8803b0c0 x1874672518164736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829867.925564:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518164736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829892 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829867.925569:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829867.925571:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829867.925574:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518164736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829868.244756:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829868.756736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829868.756852:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829868.948722:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829868.948753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829869.268749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829869.780789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829869.780861:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829869.972803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829869.972804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829870.292727:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829870.804759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829870.804821:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829870.996780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829870.996780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829871.316779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829871.828770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829871.828892:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829872.020766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829872.020766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829872.340769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829872.852762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829872.852813:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829873.044790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829873.044879:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829873.044928:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829873.044934:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829873.044938:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829873.044947:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2879 00000100:00080000:0.0:1787829873.044959:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829873.044966:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829873.045027:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518164864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829873.045058:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829873.045063:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829873.045066:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829873.045073:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2879 00000100:00080000:0.0:1787829873.045078:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829873.045082:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829873.045093:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829873.045104:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518164864:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829873.045108:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518164992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829873.045123:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829873.045126:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829873.045128:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829873.045131:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2879 00000100:00080000:0.0:1787829873.045135:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829873.045140:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829873.045144:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829873.045153:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518165120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829873.045160:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829873.045163:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829873.045165:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829873.045167:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2879 00000100:00080000:0.0:1787829873.045170:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829873.045176:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829873.045177:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829873.045189:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829873.045204:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518165248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829873.045211:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829873.045214:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829873.045215:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829873.045219:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2879 00000100:00080000:0.0:1787829873.045222:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829873.045227:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829873.045231:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829873.045235:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829873/real 1787829873] req@ffff8b2e880421c0 x1874672518164864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829873.045245:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518165376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829873.045255:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829873.045258:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518164864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829873.045261:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829873.045264:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829873.045280:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2879 00000100:00080000:0.0:1787829873.045290:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829873.045303:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829873.045306:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829873.045312:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829873.045313:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829873.045319:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518164864:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829873.045334:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518165504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829873.045345:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518164992:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829873.045360:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829873.045364:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829873.045366:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829873.045372:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2879 00000100:00080000:0.0:1787829873.045377:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829873.045381:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829873.045386:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829873.045391:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829873.045402:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518165632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829873.045414:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829873/real 1787829873] req@ffff8b2e88041a40 x1874672518164992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829873.045432:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518164992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829873.045442:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829873.045446:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829873.045450:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518164992:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829873.045457:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518165120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829873.045480:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829873.045490:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829873/real 1787829873] req@ffff8b2e88040f00 x1874672518165120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829873.045513:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518165120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829873.045521:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829873.045525:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829873.045528:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518165120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829873.045534:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518165248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829873.045553:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829873.045564:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829873/real 1787829873] req@ffff8b2e88042940 x1874672518165248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829873.045575:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518165248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829873.045583:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829873.045587:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829873.045590:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518165248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829873.045596:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518165376:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829873.045612:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829873.045623:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829873/real 1787829873] req@ffff8b2e88043480 x1874672518165376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829873.045633:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518165376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829873.045642:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829873.045667:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829873.045671:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518165376:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829873.045691:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518165504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829873.045710:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829873.045722:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829873/real 1787829873] req@ffff8b2e880430c0 x1874672518165504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829873.045735:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518165504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829873.045742:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829873.045745:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829873.045748:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518165504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829873.045754:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518165632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829873.045771:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829873.045781:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829873/real 1787829873] req@ffff8b2e88040000 x1874672518165632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829873.045793:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518165632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829898 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829873.045802:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829873.045805:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829873.045809:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518165632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829873.364777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829873.876758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829873.876858:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829874.068756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829874.068756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829874.388724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829874.900739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829874.900791:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829875.092780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829875.092780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829875.412780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829875.924769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829875.924844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829876.116769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829876.116769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829876.436729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787829876.948816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829876.948836:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787829877.140776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829877.140778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829877.460785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829877.972771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829877.972829:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829878.164824:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829878.164834:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829878.164866:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829878.164872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829878.164878:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829878.164888:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2884 00000100:00080000:0.0:1787829878.164900:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829878.164908:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829878.164992:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518165760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829878.165022:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829878.165025:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829878.165027:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829878.165031:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2884 00000100:00080000:0.0:1787829878.165036:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829878.165045:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829878.165058:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829878.165077:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518165760:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829878.165083:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518165888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829878.165100:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829878.165105:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829878.165107:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829878.165114:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2884 00000100:00080000:0.0:1787829878.165118:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829878.165123:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829878.165129:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829878.165144:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518166016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829878.165155:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829878.165160:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829878.165162:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829878.165166:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2884 00000800:00000100:3.0:1787829878.165180:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829878.165183:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829878.165192:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829878.165198:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829878.165213:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518166144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829878.165236:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829878.165240:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829878.165247:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829878.165247:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829878/real 1787829878] req@ffff8b2e880403c0 x1874672518165760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829878.165261:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2884 00000100:00080000:0.0:1787829878.165272:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829878.165281:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829878.165290:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829878.165297:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518165760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829878.165315:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829878.165317:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518166272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829878.165324:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829878.165335:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518165760:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829878.165351:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829878.165355:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829878.165359:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829878.165364:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2884 00000100:00080000:0.0:1787829878.165369:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829878.165374:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518165888:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829878.165380:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829878.165390:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829878.165414:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518166400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829878.165428:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829878.165449:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829878.165450:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829878/real 1787829878] req@ffff8b2e88040000 x1874672518165888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829878.165460:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829878.165467:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829878.165477:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2884 00000100:00080000:0.0:1787829878.165482:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829878.165487:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829878.165493:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518165888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829878.165496:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829878.165518:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829878.165523:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518166528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829878.165530:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829878.165540:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518165888:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829878.165563:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518166016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829878.165598:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829878.165612:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829878/real 1787829878] req@ffff8b2e880430c0 x1874672518166016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829878.165623:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518166016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829878.165629:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829878.165632:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829878.165635:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518166016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829878.165640:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518166144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829878.165700:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829878.165714:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829878/real 1787829878] req@ffff8b2e88043480 x1874672518166144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829878.165727:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518166144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829878.165736:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829878.165739:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829878.165743:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518166144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829878.165750:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518166272:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829878.165772:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829878.165785:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829878/real 1787829878] req@ffff8b2e88042940 x1874672518166272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829878.165796:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518166272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829878.165805:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829878.165808:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829878.165812:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518166272:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829878.165830:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518166400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829878.165852:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829878.165864:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829878/real 1787829878] req@ffff8b2e88040f00 x1874672518166400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829878.165877:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518166400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829878.165885:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829878.165888:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829878.165891:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518166400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829878.165897:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518166528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829878.165915:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829878.165925:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829878/real 1787829878] req@ffff8b2e88041a40 x1874672518166528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829878.165937:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518166528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829903 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829878.165945:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829878.165948:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829878.165951:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518166528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829878.484730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829878.996771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829878.996837:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829879.188747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829879.188747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829879.508784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829880.020759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829880.020797:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829880.212741:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829880.212741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829880.532723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829881.044751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829881.044801:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829881.236731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829881.236751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829881.556745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829882.068770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829882.068825:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829882.260758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829882.260758:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829882.580767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829883.092819:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829883.092867:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787829883.284779:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.284792:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787829883.284796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829883.284797:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829883.284812:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2889 00000100:00080000:1.0:1787829883.284825:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.284837:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829883.284848:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829883.284882:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518166656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829883.284915:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.284919:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829883.284936:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829883.284943:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2889 00000100:00080000:1.0:1787829883.284947:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.284952:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829883.284963:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829883.284970:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518166656:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829883.284982:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518166784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829883.285011:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.285015:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829883.285017:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829883.285022:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2889 00000100:00080000:1.0:1787829883.285046:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.285052:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829883.285058:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829883.285064:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829883.285079:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518166912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829883.285091:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.285094:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829883.285097:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829883.285102:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2889 00000100:00080000:1.0:1787829883.285107:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.285112:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829883.285119:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829883.285130:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff8b2e88075e00 x1874672518166656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829883.285136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518167040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829883.285180:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.285184:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829883.285189:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829883.285189:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518166656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829883.285206:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2889 00000100:00080000:1.0:1787829883.285214:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.285222:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829883.285222:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829883.285237:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787829883.285239:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829883.285245:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518166656:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829883.285268:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518167168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829883.285287:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518166784:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829883.285288:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.285305:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829883.285309:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829883.285316:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2889 00000100:00080000:1.0:1787829883.285324:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.285329:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829883.285335:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829883.285339:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829883.285358:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518167296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829883.285365:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff8b2e880761c0 x1874672518166784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829883.285382:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829883.285398:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829883.285408:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829883.285409:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518166784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829883.285422:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2889 00000100:00080000:1.0:1787829883.285427:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829883.285437:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829883.285450:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829883.285453:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829883.285462:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829883.285469:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518167424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829883.285470:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518166784:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829883.285497:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518166912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829883.285527:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829883.285539:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff8b2e88074000 x1874672518166912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829883.285561:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518166912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829883.285571:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829883.285575:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829883.285579:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518166912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829883.285586:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518167040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829883.285604:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829883.285615:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff8b2e88074b40 x1874672518167040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829883.285628:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518167040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829883.285637:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829883.285640:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829883.285644:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518167040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829883.285713:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518167168:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829883.285735:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829883.285748:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff8b2e880770c0 x1874672518167168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829883.285760:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518167168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829883.285776:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829883.285779:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829883.285783:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518167168:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829883.285789:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518167296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829883.285811:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829883.285823:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff8b2e88077840 x1874672518167296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829883.285837:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518167296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829883.285845:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829883.285847:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829883.285850:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518167296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829883.285855:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518167424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829883.285872:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829883.285880:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829883/real 1787829883] req@ffff8b2e88076d00 x1874672518167424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829883.285887:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518167424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829908 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829883.285893:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829883.285895:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829883.285898:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518167424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829883.604753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829884.116777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829884.116861:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829884.308769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829884.308788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829884.628789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829885.140723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829885.140795:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829885.332780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829885.332784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829885.652793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829886.164783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829886.164805:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829886.356747:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829886.356813:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829886.676794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829887.188755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829887.188837:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829887.380827:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829887.380828:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829887.700787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829888.212808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829888.212841:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829888.404748:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829888.404763:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829888.404768:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829888.404776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829888.404781:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2894 00000100:00100000:3.0:1787829888.404782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829888.404802:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829888.404812:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829888.404859:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518167552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829888.404904:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829888.404909:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829888.404912:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829888.404919:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2894 00000100:00080000:0.0:1787829888.404925:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829888.404937:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518167552:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829888.404946:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829888.404961:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829888.404984:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518167680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829888.404994:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829888.404998:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829888.405000:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829888.405008:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829888.405013:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2894 00000100:00080000:0.0:1787829888.405018:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829888.405024:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829888.405032:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829888.405049:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518167808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829888.405061:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829888.405067:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff8b2e880421c0 x1874672518167552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829888.405068:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829888.405074:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829888.405089:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2894 00000100:00080000:0.0:1787829888.405093:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829888.405095:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518167552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829888.405102:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829888.405116:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829888.405120:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829888.405125:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829888.405131:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518167936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829888.405131:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518167552:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829888.405153:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829888.405156:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829888.405159:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829888.405163:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2894 00000100:00080000:0.0:1787829888.405166:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829888.405172:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518167680:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829888.405177:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829888.405181:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829888.405196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518168064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829888.405208:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829888.405218:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829888.405219:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829888.405226:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829888.405233:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2894 00000100:00080000:0.0:1787829888.405237:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829888.405243:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829888.405245:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff8b2e88040b40 x1874672518167680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829888.405248:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829888.405267:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518168192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829888.405295:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829888.405296:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518167680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829888.405313:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829888.405319:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829888.405331:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2894 00000100:00080000:3.0:1787829888.405333:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829888.405337:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829888.405338:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829888.405346:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518167680:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829888.405350:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829888.405356:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518167808:192.168.203.101@tcp:8:kworker.0 00000008:00100000:0.0:1787829888.405359:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829888.405383:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829888.405389:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518168320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829888.405397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff8b2e880412c0 x1874672518167808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829888.405430:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518167808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829888.405438:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829888.405441:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829888.405444:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518167808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829888.405449:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518167936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829888.405461:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829888.405468:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff8b2e88043840 x1874672518167936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829888.405476:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518167936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829888.405485:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829888.405488:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829888.405491:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518167936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829888.405508:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518168064:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829888.405529:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829888.405542:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff8b2e88040780 x1874672518168064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829888.405554:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518168064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829888.405560:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829888.405562:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829888.405564:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518168064:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829888.405569:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518168192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829888.405584:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829888.405591:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff8b2e88042580 x1874672518168192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829888.405598:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518168192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829888.405604:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829888.405606:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829888.405608:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518168192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829888.405613:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518168320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829888.405622:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829888.405628:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829888/real 1787829888] req@ffff8b2e88042d00 x1874672518168320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829888.405634:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518168320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829913 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829888.405639:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829888.405641:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829888.405643:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518168320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829888.724775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829889.236791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829889.236828:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829889.428789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829889.428823:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829889.748743:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829890.260764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829890.260810:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829890.452711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829890.452785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829890.772760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829891.284787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829891.284854:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829891.476784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829891.476818:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829891.796750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829892.308787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829892.308850:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829892.500800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829892.500802:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829892.820784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829893.332793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829893.332902:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829893.524720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829893.524727:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829893.524755:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829893.524765:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829893.524770:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829893.524780:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2899 00000100:00080000:0.0:1787829893.524793:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829893.524803:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829893.524860:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518168448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829893.524895:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829893.524899:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829893.524901:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829893.524906:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2899 00000100:00080000:0.0:1787829893.524910:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829893.524915:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829893.524925:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829893.524933:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518168448:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829893.524943:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518168576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829893.524955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829893.524958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829893.524960:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829893.524963:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2899 00000100:00080000:0.0:1787829893.524967:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829893.524971:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829893.524976:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829893.524985:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518168704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829893.524993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829893.524996:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829893.525001:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787829893.525007:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829893.525010:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2899 00000100:00080000:0.0:1787829893.525016:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829893.525019:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829893.525023:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829893.525033:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518168832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829893.525041:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829893.525043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829893.525045:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829893.525049:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2899 00000100:00080000:0.0:1787829893.525051:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829893.525054:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829893.525059:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829893.525066:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518168960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829893.525073:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829893.525078:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829893.525080:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff8b2e88041e00 x1874672518168448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829893.525081:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829893.525094:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2899 00000100:00080000:0.0:1787829893.525109:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829893.525114:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829893.525117:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829893.525125:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518168448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829893.525130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518169088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829893.525169:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829893.525172:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829893.525174:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829893.525175:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829893.525184:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2899 00000100:00080000:0.0:1787829893.525189:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829893.525189:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829893.525200:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829893.525202:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518168448:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829893.525206:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829893.525219:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518169216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829893.525234:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518168576:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829893.525268:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829893.525279:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff8b2e88042d00 x1874672518168576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829893.525289:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518168576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829893.525300:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829893.525303:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829893.525305:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518168576:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829893.525310:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518168704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829893.525328:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829893.525336:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff8b2e88042580 x1874672518168704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829893.525344:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518168704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829893.525350:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829893.525352:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829893.525354:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518168704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829893.525359:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518168832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829893.525370:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829893.525377:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff8b2e88040780 x1874672518168832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829893.525384:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518168832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829893.525390:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829893.525392:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829893.525393:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518168832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829893.525398:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518168960:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829893.525407:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829893.525413:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff8b2e88043840 x1874672518168960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829893.525421:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518168960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829893.525426:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829893.525428:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829893.525430:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518168960:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829893.525434:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518169088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829893.525444:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829893.525450:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff8b2e880412c0 x1874672518169088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829893.525458:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518169088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829893.525463:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829893.525465:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829893.525467:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518169088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829893.525471:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518169216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829893.525481:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829893.525486:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829893/real 1787829893] req@ffff8b2e88040b40 x1874672518169216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829893.525493:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518169216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829918 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829893.525499:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829893.525501:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829893.525502:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518169216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829893.844765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829894.356726:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829894.356867:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829894.548748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829894.548748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829894.868762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829895.380767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829895.380794:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829895.572778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829895.572778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829895.892729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829896.404765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829896.404840:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829896.596767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829896.596767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829896.916772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829897.428768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829897.428845:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829897.620773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829897.620773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829897.940784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829898.452774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829898.452829:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829898.644744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829898.644744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829898.644745:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829898.644756:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829898.644761:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829898.644771:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2905 00000100:00080000:0.0:1787829898.644783:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829898.644791:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829898.644851:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518169344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829898.644883:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829898.644887:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829898.644890:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829898.644894:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2905 00000100:00080000:0.0:1787829898.644898:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829898.644903:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829898.644914:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829898.644929:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518169472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829898.644933:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518169344:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829898.644954:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829898.644959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829898.644961:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829898.644966:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2905 00000100:00080000:0.0:1787829898.644981:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829898.644986:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829898.644992:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829898.645006:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518169600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829898.645011:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829898.645023:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829898.645027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829898.645029:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829898.645035:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2905 00000100:00080000:0.0:1787829898.645039:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829898.645044:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829898.645049:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829898.645063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518169728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829898.645069:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829898] req@ffff8b2e880421c0 x1874672518169344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829898.645085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829898.645091:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829898.645094:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829898.645099:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518169344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829898.645103:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2905 00000100:00080000:0.0:1787829898.645111:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829898.645118:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829898.645119:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829898.645124:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829898.645127:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829898.645134:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518169344:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829898.645142:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518169856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829898.645163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829898.645169:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:3.0:1787829898.645169:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518169472:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829898.645174:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829898.645181:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2905 00000100:00080000:0.0:1787829898.645187:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829898.645191:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829898.645197:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829898.645205:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829898.645212:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518169984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829898.645224:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829898] req@ffff8b2e88041a40 x1874672518169472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829898.645258:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518169472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829898.645261:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829898.645271:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787829898.645277:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829898.645282:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829898.645284:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829898.645290:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518169472:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829898.645295:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2905 00000100:00080000:0.0:1787829898.645302:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829898.645302:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518169600:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829898.645311:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829898.645318:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829898.645329:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829898.645343:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518170112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829898.645345:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829898] req@ffff8b2e88040f00 x1874672518169600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829898.645383:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518169600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829898.645392:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829898.645395:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829898.645398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518169600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829898.645403:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518169728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829898.645419:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829898.645427:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829898] req@ffff8b2e88042940 x1874672518169728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829898.645434:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518169728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829898.645440:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829898.645442:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829898.645444:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518169728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829898.645447:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518169856:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829898.645457:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829898.645463:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829898] req@ffff8b2e88043480 x1874672518169856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829898.645470:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518169856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829898.645479:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829898.645481:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829898.645483:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518169856:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829898.645496:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518169984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829898.645506:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829898.645512:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829898] req@ffff8b2e880430c0 x1874672518169984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829898.645519:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518169984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829898.645524:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829898.645526:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829898.645528:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518169984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829898.645531:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518170112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829898.645540:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829898.645546:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829898/real 1787829898] req@ffff8b2e88040000 x1874672518170112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829898.645553:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518170112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829923 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829898.645558:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829898.645560:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829898.645563:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518170112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829898.964766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829899.476752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829899.476833:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829899.668754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829899.668754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829899.988747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829900.500716:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829900.500910:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829900.692727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829900.692755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829901.012745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829901.524760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829901.524774:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829901.716753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829901.716753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829902.036750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829902.548769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829902.548852:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829902.740921:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829902.740924:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829903.060713:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829903.572767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829903.572922:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829903.764746:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829903.764754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829903.764758:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829903.764765:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787829903.764774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829903.764775:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2910 00000100:00080000:0.0:1787829903.764797:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829903.764806:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829903.764843:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518170240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829903.764865:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829903.764868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829903.764870:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829903.764874:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2910 00000100:00080000:0.0:1787829903.764878:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829903.764882:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829903.764892:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829903.764893:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518170240:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829903.764907:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518170368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829903.764918:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829903.764921:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829903.764922:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829903.764926:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2910 00000100:00080000:0.0:1787829903.764929:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829903.764932:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829903.764936:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:3.0:1787829903.764942:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829903.764948:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518170496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829903.764957:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829903.764960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829903.764962:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829903.764965:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2910 00000100:00080000:0.0:1787829903.764968:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829903.764972:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829903.764976:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829903.764982:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff8b2e880403c0 x1874672518170240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829903.764986:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518170624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829903.765002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829903.765005:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829903.765007:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829903.765008:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518170240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829903.765011:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2910 00000100:00080000:0.0:1787829903.765016:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829903.765021:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829903.765024:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829903.765025:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829903.765031:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829903.765035:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518170752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829903.765037:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518170240:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829903.765050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829903.765052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829903.765054:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829903.765059:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2910 00000100:00100000:3.0:1787829903.765061:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518170368:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829903.765064:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829903.765070:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829903.765075:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829903.765082:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829903.765088:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518170880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829903.765097:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829903.765098:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff8b2e88040000 x1874672518170368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829903.765103:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829903.765106:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829903.765112:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2910 00000100:00080000:0.0:1787829903.765117:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829903.765121:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829903.765124:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829903.765125:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518170368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829903.765134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518171008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829903.765135:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829903.765142:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829903.765148:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518170368:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829903.765153:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518170496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829903.765169:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829903.765176:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff8b2e880430c0 x1874672518170496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829903.765183:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518170496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829903.765188:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829903.765190:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829903.765192:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518170496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829903.765195:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518170624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829903.765203:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829903.765208:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff8b2e88043480 x1874672518170624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829903.765214:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518170624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829903.765218:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829903.765220:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829903.765221:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518170624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829903.765224:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518170752:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829903.765232:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829903.765238:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff8b2e88042940 x1874672518170752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829903.765244:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518170752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829903.765248:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829903.765250:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829903.765251:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518170752:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829903.765264:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518170880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829903.765281:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829903.765292:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff8b2e88040f00 x1874672518170880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829903.765303:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518170880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829903.765309:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829903.765311:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829903.765313:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518170880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829903.765316:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518171008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829903.765329:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829903.765335:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829903/real 1787829903] req@ffff8b2e88041a40 x1874672518171008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829903.765346:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518171008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829928 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829903.765350:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829903.765352:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829903.765354:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518171008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829904.084774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829904.596734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829904.596795:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829904.788783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829904.788784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829905.108723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829905.620784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829905.620845:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829905.812730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829905.812743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829906.132778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829906.644751:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829906.644844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829906.836789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829906.836796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829907.156768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829907.668770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829907.668810:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829907.860711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829907.860764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829908.180797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829908.692777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829908.692854:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829908.884795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829908.884801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829908.884843:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829908.884850:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829908.884854:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829908.884864:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2915 00000100:00080000:2.0:1787829908.884874:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829908.884881:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829908.884920:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518171136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829908.884955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829908.884958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829908.884960:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829908.884964:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2915 00000100:00080000:2.0:1787829908.884968:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829908.884972:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829908.884982:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829908.884996:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518171264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829908.885016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829908.885018:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518171136:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829908.885026:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829908.885030:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829908.885036:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2915 00000100:00080000:2.0:1787829908.885039:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829908.885044:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829908.885049:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829908.885060:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518171392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829908.885067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829908.885069:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829908.885070:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829908.885073:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2915 00000100:00080000:2.0:1787829908.885093:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829908.885096:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829908.885100:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829908.885106:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518171520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829908.885118:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829908.885119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829908.885127:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829908.885129:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829908.885132:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2915 00000100:00080000:2.0:1787829908.885134:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829908.885137:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829908.885139:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829908.885152:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518171648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829908.885158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829908.885160:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829908.885162:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829908.885167:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2915 00000100:00080000:2.0:1787829908.885170:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829908.885174:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829908.885178:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829908.885191:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff8b2e8801e940 x1874672518171136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829908.885201:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518171776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829908.885231:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829908.885235:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518171136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829908.885237:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829908.885242:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829908.885259:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2915 00000100:00080000:2.0:1787829908.885268:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829908.885272:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829908.885276:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829908.885277:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829908.885290:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829908.885296:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518171904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829908.885304:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518171136:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829908.885348:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518171264:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829908.885383:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829908.885397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff8b2e8801e580 x1874672518171264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829908.885409:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518171264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.885418:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829908.885428:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829908.885432:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518171264:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829908.885438:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518171392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829908.885456:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829908.885471:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff8b2e8801c000 x1874672518171392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829908.885483:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518171392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.885491:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829908.885494:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829908.885497:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518171392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829908.885504:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518171520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829908.885519:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829908.885528:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff8b2e8801cf00 x1874672518171520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829908.885552:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518171520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.885560:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829908.885563:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829908.885566:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518171520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829908.885572:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518171648:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829908.885588:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829908.885597:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff8b2e8801f480 x1874672518171648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829908.885608:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518171648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.885616:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829908.885619:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829908.885622:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518171648:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829908.885627:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518171776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829908.885642:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829908.885677:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff8b2e8801fc00 x1874672518171776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829908.885689:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518171776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.885702:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829908.885705:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829908.885709:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518171776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829908.885715:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518171904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829908.885740:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829908.885764:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829908/real 1787829908] req@ffff8b2e8801da40 x1874672518171904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829908.885776:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518171904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829933 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829908.885784:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829908.885793:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829908.885796:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518171904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829909.204782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829909.716769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829909.716841:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829909.908782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829909.908783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829910.228770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787829910.740899:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829910.740935:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787829910.932744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829910.932778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829911.252791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829911.764760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829911.764817:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829911.957805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829911.958043:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829912.276798:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829912.788775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829912.788830:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829912.980743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829912.980775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829913.300790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829913.812734:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829913.812790:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829914.004783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829914.004787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829914.004814:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829914.004822:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829914.004827:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829914.004840:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2920 00000100:00080000:1.0:1787829914.004853:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829914.004862:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829914.004904:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518172032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829914.004932:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829914.004936:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829914.004937:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829914.004942:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2920 00000100:00080000:1.0:1787829914.004945:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829914.004948:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829914.004966:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829914.004980:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518172160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829914.004987:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518172032:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829914.004993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829914.005000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829914.005002:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829914.005006:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2920 00000100:00080000:1.0:1787829914.005009:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829914.005013:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829914.005018:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829914.005030:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518172288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829914.005038:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829914.005042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829914.005044:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829914.005048:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2920 00000100:00080000:1.0:1787829914.005051:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829914.005055:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829914.005060:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829914.005073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518172416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829914.005073:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829914.005084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829914.005088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829914.005090:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829914.005094:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2920 00000100:00080000:1.0:1787829914.005100:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829914.005104:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829914.005108:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829914.005120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518172544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829914.005127:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829914/real 1787829914] req@ffff8b2e88076580 x1874672518172032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829914.005140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829914.005144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829914.005147:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829914.005149:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518172032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829914.005154:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2920 00000100:00080000:1.0:1787829914.005163:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829914.005167:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829914.005170:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829914.005175:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:1.0:1787829914.005176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829914.005181:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518172032:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829914.005190:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518172672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829914.005197:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829914.005200:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829914.005201:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829914.005205:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2920 00000100:00080000:1.0:1787829914.005209:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829914.005210:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518172160:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829914.005217:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829914.005224:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829914.005247:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518172800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829914.005250:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829914.005267:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829914/real 1787829914] req@ffff8b2e88076d00 x1874672518172160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829914.005282:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518172160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829914.005291:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829914.005294:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829914.005297:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518172160:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829914.005303:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518172288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829914.005324:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829914.005345:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829914/real 1787829914] req@ffff8b2e88077840 x1874672518172288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829914.005363:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518172288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829914.005368:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829914.005370:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829914.005373:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518172288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829914.005376:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518172416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829914.005391:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829914.005397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829914/real 1787829914] req@ffff8b2e880770c0 x1874672518172416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829914.005404:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518172416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829914.005409:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829914.005411:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829914.005412:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518172416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829914.005416:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518172544:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829914.005433:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829914.005440:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829914/real 1787829914] req@ffff8b2e88074b40 x1874672518172544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829914.005446:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518172544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829914.005456:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829914.005458:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829914.005460:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518172544:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829914.005463:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518172672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829914.005473:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829914.005478:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829914/real 1787829914] req@ffff8b2e88074000 x1874672518172672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829914.005484:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518172672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829914.005489:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829914.005490:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829914.005492:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518172672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829914.005510:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518172800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829914.005521:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829914.005527:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829914/real 1787829914] req@ffff8b2e880761c0 x1874672518172800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829914.005534:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518172800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829939 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829914.005539:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829914.005540:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829914.005543:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518172800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829914.324796:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829914.836758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829914.836854:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829915.028784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829915.028787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829915.348750:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829915.860779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829915.860844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829916.052787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829916.052791:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829916.372784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829916.884786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829916.884870:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829917.076763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829917.076763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829917.396741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829917.908803:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829917.908871:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829918.100757:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829918.100760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829918.420790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829918.932794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829918.932854:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829919.124764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829919.124779:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829919.124787:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:2.0:1787829919.124787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829919.124795:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829919.124810:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2925 00000100:00080000:0.0:1787829919.124823:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829919.124832:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829919.124877:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518172928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829919.124907:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829919.124911:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829919.124913:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829919.124918:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2925 00000100:00080000:0.0:1787829919.124922:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829919.124926:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829919.124937:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829919.124943:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518172928:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829919.124954:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518173056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829919.124966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829919.124970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829919.124972:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829919.124976:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2925 00000100:00080000:0.0:1787829919.124980:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829919.124984:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829919.124989:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829919.125000:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518173184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829919.125009:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829919.125011:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829919.125016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829919.125018:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829919.125023:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2925 00000100:00080000:0.0:1787829919.125027:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829919.125031:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829919.125037:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829919.125049:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518173312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829919.125054:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829919/real 1787829919] req@ffff8b2e880421c0 x1874672518172928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829919.125067:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829919.125072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829919.125075:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829919.125076:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518172928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829919.125084:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2925 00000100:00080000:3.0:1787829919.125088:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829919.125090:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829919.125093:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829919.125099:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829919.125099:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518172928:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829919.125108:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829919.125124:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518173440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829919.125124:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518173056:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829919.125140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829919.125145:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000800:00000100:3.0:1787829919.125147:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829919.125149:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829919.125156:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2925 00000100:00100000:3.0:1787829919.125159:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829919/real 1787829919] req@ffff8b2e88040b40 x1874672518173056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829919.125162:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829919.125171:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829919.125174:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518173056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829919.125178:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829919.125183:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829919.125187:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829919.125190:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518173056:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829919.125195:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518173184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829919.125203:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518173568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829919.125210:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829919.125219:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829919.125220:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829919/real 1787829919] req@ffff8b2e880412c0 x1874672518173184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829919.125227:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829919.125231:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829919.125235:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518173184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829919.125239:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2925 00000100:00080000:0.0:1787829919.125247:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787829919.125248:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829919.125253:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829919.125255:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829919.125258:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518173184:192.168.203.101@tcp:8:kworker.0 00000008:00100000:0.0:1787829919.125264:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829919.125265:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518173312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829919.125282:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518173696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829919.125285:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829919.125294:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829919/real 1787829919] req@ffff8b2e88043840 x1874672518173312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829919.125312:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518173312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829919.125317:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829919.125320:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829919.125322:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518173312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829919.125331:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518173440:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829919.125342:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829919.125349:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829919/real 1787829919] req@ffff8b2e88040780 x1874672518173440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829919.125356:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518173440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829919.125361:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829919.125363:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829919.125365:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518173440:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829919.125368:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518173568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829919.125375:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829919.125380:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829919/real 1787829919] req@ffff8b2e88042580 x1874672518173568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829919.125387:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518173568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829919.125391:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829919.125393:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829919.125394:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518173568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829919.125398:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518173696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829919.125406:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829919.125411:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829919/real 1787829919] req@ffff8b2e88042d00 x1874672518173696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829919.125421:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518173696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829944 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829919.125426:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829919.125428:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829919.125429:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518173696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829919.444780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829919.956811:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829919.956858:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829920.148724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829920.148732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829920.468736:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829920.980815:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829920.980863:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829921.172749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829921.172771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829921.492790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829922.004727:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829922.004805:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829922.196719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829922.196749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829922.516747:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829923.028793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829923.028827:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829923.220779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829923.220781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829923.540772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829924.052736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829924.052797:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829924.244755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829924.244755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829924.244761:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829924.244770:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829924.244775:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829924.244787:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2930 00000100:00080000:0.0:1787829924.244800:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829924.244808:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829924.244852:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518173824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829924.244879:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829924.244883:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829924.244885:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829924.244891:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2930 00000100:00080000:0.0:1787829924.244895:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829924.244900:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829924.244910:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518173824:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829924.244911:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829924.244931:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518173952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829924.244942:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829924.244946:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829924.244948:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829924.244952:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2930 00000100:00080000:0.0:1787829924.244957:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829924.244961:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829924.244967:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829924.244980:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518174080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829924.244990:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829924.244993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829924.244995:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829924.245000:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2930 00000100:00080000:0.0:1787829924.245006:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829924.245010:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829924.245015:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829924.245015:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518173952:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829924.245034:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518174208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829924.245043:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518174080:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829924.245045:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829924.245050:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829924.245053:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829924.245059:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2930 00000100:00080000:0.0:1787829924.245063:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829924.245065:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518174208:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829924.245071:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829924.245076:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829924.245090:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518174336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829924.245105:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829924.245109:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829924.245112:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829924.245117:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2930 00000100:00080000:0.0:1787829924.245121:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829924.245122:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518174336:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829924.245128:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829924.245135:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829924.245149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518174464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829924.245169:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829924.245173:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829924.245175:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829924.245179:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2930 00000100:00080000:0.0:1787829924.245182:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829924.245187:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829924.245192:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518174464:192.168.203.101@tcp:8:kworker.0 00000008:00100000:0.0:1787829924.245193:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829924.245210:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518174592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829924.245230:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518174592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829924.564734:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829925.076761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829925.076844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829925.268734:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829925.268780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829925.588751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829926.100814:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829926.100887:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829926.292777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829926.292838:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829926.612754:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829927.124739:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829927.124778:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829927.316713:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829927.316841:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:3.0:1787829927.316939:0:2336:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:3.0:1787829927.317033:0:2336:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787829927.317050:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829927.317096:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829927.317106:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829927.317113:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829927.317121:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829927.317127:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787829927.317134:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787829927.317206:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829927] req@ffff8b2e88040b40 x1874672518174592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829927.317231:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518174592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829927.317244:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829927.317250:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829927.317255:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518174592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829927.317262:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829927] req@ffff8b2e880412c0 x1874672518174464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829927.317275:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518174464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829927.317283:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829927.317287:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829927.317290:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518174464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829927.317297:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829927] req@ffff8b2e88043840 x1874672518174336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829927.317308:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518174336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829927.317316:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829927.317319:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829927.317322:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518174336:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829927.317328:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829927] req@ffff8b2e88040780 x1874672518174208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829927.317346:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518174208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829927.317354:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829927.317358:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829927.317361:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518174208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829927.317367:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829927] req@ffff8b2e88042d00 x1874672518173952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829927.317377:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518173952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829927.317386:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829927.317389:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829927.317392:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518173952:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787829927.317398:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829927] req@ffff8b2e88042580 x1874672518174080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829927.317409:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518174080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829927.317416:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829927.317420:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829927.317423:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518174080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829927.317429:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829924/real 1787829927] req@ffff8b2e88041e00 x1874672518173824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829927.317440:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518173824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829949 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829927.317448:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829927.317450:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829927.317453:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518173824:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829927.636779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829928.148728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829928.148762:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829928.340764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829928.340803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829928.660773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829929.172766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829929.172823:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829929.364842:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829929.364842:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829929.364853:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829929.364857:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829929.364867:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2935 00000100:00080000:0.0:1787829929.364877:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829929.364884:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829929.364916:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518174720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829929.364942:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829929.364945:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829929.364950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829929.364952:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829929.364959:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2935 00000100:00080000:0.0:1787829929.364963:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829929.364968:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829929.364977:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829929.364989:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518174848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829929.365000:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829929.365003:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829929.365006:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829929.365008:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518174720:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829929.365011:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2935 00000100:00080000:0.0:1787829929.365016:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829929.365021:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829929.365026:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829929.365039:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518174976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829929.365049:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829929.365052:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829929.365054:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829929.365058:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2935 00000100:00080000:0.0:1787829929.365061:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829929.365065:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829929.365070:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829929.365081:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829929.365083:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518175104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829929.365093:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829929.365097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829929.365099:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829929.365103:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2935 00000100:00080000:0.0:1787829929.365106:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829929.365111:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829929.365116:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829929.365128:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518175232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829929.365132:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff8b2e88041e00 x1874672518174720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829929.365143:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829929.365151:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829929.365155:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829929.365161:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2935 00000100:00080000:0.0:1787829929.365164:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829929.365168:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518174720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829929.365170:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829929.365177:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829929.365189:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829929.365194:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518175360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829929.365196:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829929.365204:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518174720:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829929.365213:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829929.365217:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829929.365219:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829929.365224:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2935 00000100:00080000:0.0:1787829929.365227:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829929.365231:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829929.365234:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518174848:192.168.203.101@tcp:38:kworker.0 00000008:00100000:0.0:1787829929.365237:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829929.365251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518175488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829929.365262:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829929.365277:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff8b2e88042580 x1874672518174848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829929.365297:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518174848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829929.365306:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829929.365309:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829929.365312:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518174848:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829929.365318:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518174976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829929.365340:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829929.365351:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff8b2e88042d00 x1874672518174976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829929.365365:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518174976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829929.365373:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829929.365377:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829929.365394:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518174976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829929.365400:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518175104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829929.365422:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829929.365434:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff8b2e88040780 x1874672518175104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829929.365444:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518175104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829929.365456:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829929.365458:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829929.365461:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518175104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829929.365467:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518175232:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829929.365497:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829929.365508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff8b2e88043840 x1874672518175232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829929.365519:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518175232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829929.365528:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829929.365531:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829929.365534:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518175232:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829929.365691:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518175360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829929.365714:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829929.365727:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff8b2e880412c0 x1874672518175360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829929.365746:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518175360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829929.365754:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829929.365757:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829929.365760:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518175360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829929.365774:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518175488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829929.365796:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829929.365807:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829929/real 1787829929] req@ffff8b2e88040b40 x1874672518175488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829929.365816:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518175488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829954 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829929.365830:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829929.365832:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829929.365835:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518175488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829929.684759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829930.196780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829930.196843:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829930.388745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829930.388818:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829930.708764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787829931.220736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829931.220825:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829931.412766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829931.412766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829931.732761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829932.244955:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829932.244963:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787829932.436756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829932.436756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829932.757508:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829933.268737:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829933.268782:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829933.460777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829933.460865:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829933.780749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829934.292817:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829934.293328:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829934.484752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829934.484763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829934.484779:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829934.484785:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829934.484789:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829934.484796:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2940 00000100:00080000:0.0:1787829934.484805:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829934.484810:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829934.484842:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518175616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829934.484859:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829934.484861:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829934.484862:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829934.484864:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2940 00000100:00080000:0.0:1787829934.484867:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829934.484870:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829934.484878:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829934.484883:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518175616:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829934.484888:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518175744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829934.484896:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829934.484898:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829934.484899:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829934.484901:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2940 00000100:00080000:0.0:1787829934.484903:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829934.484905:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829934.484908:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829934.484916:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518175872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829934.484921:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829934.484923:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829934.484925:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829934.484926:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2940 00000100:00080000:0.0:1787829934.484928:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829934.484930:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787829934.484934:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787829934.484935:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829934.484943:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518176000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829934.484949:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829934.484952:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829934.484954:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829934.484956:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2940 00000100:00080000:0.0:1787829934.484959:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829934.484961:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829934.484964:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829934.484971:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518176128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829934.484975:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829934.484977:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829934.484978:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829934.484983:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2940 00000100:00100000:3.0:1787829934.484983:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff8b2e880421c0 x1874672518175616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829934.484984:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829934.484990:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829934.484996:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829934.485004:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518176256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829934.485007:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518175616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829934.485018:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829934.485024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829934.485027:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829934.485030:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2940 00000100:00080000:3.0:1787829934.485031:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829934.485032:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829934.485038:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829934.485041:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:0.0:1787829934.485043:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829934.485050:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518175616:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829934.485052:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518176384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829934.485078:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518175744:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829934.485102:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829934.485116:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff8b2e88041a40 x1874672518175744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829934.485126:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518175744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829934.485131:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829934.485133:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829934.485135:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518175744:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829934.485139:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518175872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829934.485151:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829934.485157:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff8b2e88040f00 x1874672518175872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829934.485164:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518175872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829934.485169:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829934.485171:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829934.485172:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518175872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829934.485175:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518176000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829934.485183:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829934.485188:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff8b2e88042940 x1874672518176000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829934.485195:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518176000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829934.485199:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829934.485201:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829934.485202:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518176000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829934.485205:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518176128:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829934.485214:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829934.485219:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff8b2e88043480 x1874672518176128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829934.485231:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518176128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829934.485239:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829934.485241:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829934.485243:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518176128:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829934.485246:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518176256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829934.485256:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829934.485261:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff8b2e880430c0 x1874672518176256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829934.485267:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518176256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829934.485272:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829934.485274:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829934.485276:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518176256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829934.485279:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518176384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829934.485287:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829934.485292:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829934/real 1787829934] req@ffff8b2e88040000 x1874672518176384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829934.485298:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518176384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829959 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829934.485302:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829934.485304:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829934.485305:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518176384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829934.804738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829935.316776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829935.317062:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829935.509167:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829935.509644:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829935.828768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829936.340756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829936.340817:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829936.532730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829936.532741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829936.852718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829937.364759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829937.364827:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829937.556730:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829937.556732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829937.876775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829938.388752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829938.388810:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829938.580759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829938.580764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829938.900771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829939.412771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829939.412824:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829939.604734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829939.604735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829939.604796:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.604805:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829939.604809:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.604818:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2946 00000100:00080000:2.0:1787829939.604827:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829939.604834:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829939.604872:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518176512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829939.604906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.604910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829939.604912:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.604915:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2946 00000100:00080000:2.0:1787829939.604919:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829939.604923:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829939.604939:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829939.604954:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518176640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829939.604964:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518176512:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829939.604966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.604977:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829939.604979:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.604983:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2946 00000100:00080000:2.0:1787829939.604987:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829939.604991:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829939.604996:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829939.605008:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518176768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829939.605017:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.605020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829939.605021:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.605027:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2946 00000800:00000100:3.0:1787829939.605029:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829939.605031:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829939.605037:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829939.605041:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829939.605052:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518176896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829939.605061:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.605065:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829939.605067:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.605071:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2946 00000100:00080000:2.0:1787829939.605074:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829939.605078:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829939.605082:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829939.605091:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff8b2e8801ed00 x1874672518176512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829939.605095:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518177024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829939.605121:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.605125:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829939.605127:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.605132:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2946 00000100:00100000:3.0:1787829939.605133:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518176512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829939.605136:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829939.605142:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829939.605150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829939.605152:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829939.605158:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829939.605164:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518177152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829939.605165:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518176512:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787829939.605179:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829939.605183:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829939.605185:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829939.605188:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2946 00000100:00080000:2.0:1787829939.605192:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829939.605196:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518176640:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787829939.605199:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829939.605207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829939.605219:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518177280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829939.605241:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829939.605266:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff8b2e8801c780 x1874672518176640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829939.605280:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518176640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829939.605289:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829939.605292:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829939.605296:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518176640:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829939.605302:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518176768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829939.605330:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829939.605342:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff8b2e8801f0c0 x1874672518176768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829939.605360:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518176768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829939.605368:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829939.605371:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829939.605374:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518176768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829939.605382:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518176896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829939.605401:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829939.605411:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff8b2e8801d680 x1874672518176896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829939.605422:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518176896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829939.605431:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829939.605434:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829939.605437:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518176896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829939.605443:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518177024:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829939.605461:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829939.605470:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff8b2e8801c3c0 x1874672518177024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829939.605481:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518177024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829939.605495:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829939.605498:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829939.605501:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518177024:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829939.605507:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518177152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829939.605525:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829939.605556:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff8b2e8801cb40 x1874672518177152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829939.605568:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518177152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829939.605575:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829939.605578:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829939.605581:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518177152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829939.605608:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518177280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829939.605629:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829939.605640:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829939/real 1787829939] req@ffff8b2e8801de00 x1874672518177280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829939.605689:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518177280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829964 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829939.605697:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829939.605700:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829939.605704:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518177280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829939.924777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829940.436768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829940.436880:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829940.628748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829940.628748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829940.948775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829941.460765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829941.460830:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829941.652759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829941.652759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829941.973662:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829942.484753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829942.484842:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829942.676788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829942.676790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829942.996728:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829943.508722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829943.508793:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829943.700770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829943.700857:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829944.020717:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829944.532765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829944.532832:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829944.724756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787829944.724756:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787829944.724756:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787829944.724767:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787829944.724771:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829944.724781:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2951 00000100:00080000:2.0:1787829944.724792:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829944.724800:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787829944.724835:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518177408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829944.724859:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829944.724862:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829944.724864:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829944.724868:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2951 00000100:00080000:2.0:1787829944.724872:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829944.724876:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829944.724885:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829944.724890:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518177408:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829944.724898:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518177536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829944.724909:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829944.724913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829944.724914:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829944.724917:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2951 00000100:00080000:2.0:1787829944.724920:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829944.724924:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829944.724928:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829944.724938:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518177664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829944.724948:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829944.724952:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:3.0:1787829944.724954:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787829944.724955:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829944.724962:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2951 00000100:00080000:2.0:1787829944.724965:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829944.724969:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829944.724973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829944.724982:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518177792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829944.724990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829944.724993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787829944.724995:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829944.724999:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2951 00000100:00080000:2.0:1787829944.725002:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829944.725005:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787829944.725009:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829944.725015:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff8b2e8801de00 x1874672518177408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829944.725019:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518177920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829944.725040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829944.725043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787829944.725046:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829944.725046:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518177408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829944.725053:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2951 00000100:00080000:2.0:1787829944.725059:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829944.725065:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829944.725066:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:2.0:1787829944.725069:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829944.725073:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829944.725078:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518177408:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829944.725082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518178048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829944.725093:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787829944.725096:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787829944.725098:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787829944.725101:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2951 00000100:00080000:2.0:1787829944.725105:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787829944.725110:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829944.725110:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518177536:192.168.203.101@tcp:38:kworker.0 00000008:00100000:2.0:1787829944.725117:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829944.725129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518178176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829944.725149:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829944.725164:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff8b2e8801cb40 x1874672518177536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829944.725176:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518177536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829944.725185:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829944.725189:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829944.725192:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518177536:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829944.725198:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518177664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829944.725221:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829944.725239:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff8b2e8801c3c0 x1874672518177664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829944.725250:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518177664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829944.725258:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829944.725262:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829944.725270:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518177664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829944.725276:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518177792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829944.725292:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829944.725301:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff8b2e8801d680 x1874672518177792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829944.725312:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518177792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829944.725320:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829944.725323:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829944.725326:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518177792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829944.725332:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518177920:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829944.725346:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829944.725362:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff8b2e8801f0c0 x1874672518177920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829944.725373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518177920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829944.725380:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829944.725383:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829944.725387:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518177920:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829944.725393:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518178048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829944.725407:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829944.725416:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff8b2e8801c780 x1874672518178048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829944.725427:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518178048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829944.725435:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829944.725438:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829944.725442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518178048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829944.725458:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518178176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829944.725482:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829944.725492:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829944/real 1787829944] req@ffff8b2e8801ed00 x1874672518178176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829944.725523:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518178176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829969 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829944.725531:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829944.725534:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829944.725537:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518178176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829945.044812:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829945.556745:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829945.556790:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829945.748763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829945.748844:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829946.068753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829946.580756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829946.580810:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829946.772732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829946.772732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829947.092781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829947.604753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829947.604837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829947.796764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829947.796764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829948.116775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829948.628753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829948.628802:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829948.820721:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829948.820732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829949.140766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829949.652760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829949.652829:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829949.844764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829949.844765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829949.844878:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.844883:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829949.844887:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.844895:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2956 00000100:00080000:0.0:1787829949.844906:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829949.844914:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829949.844954:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518178304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829949.844976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.844978:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829949.844980:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.844983:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2956 00000100:00080000:0.0:1787829949.844986:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829949.844988:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829949.844996:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829949.845008:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518178432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829949.845020:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518178304:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829949.845021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.845031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829949.845034:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.845038:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2956 00000100:00080000:0.0:1787829949.845042:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829949.845046:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829949.845051:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829949.845066:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518178560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829949.845076:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.845081:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829949.845083:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.845086:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2956 00000100:00080000:0.0:1787829949.845090:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:3.0:1787829949.845093:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829949.845098:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829949.845105:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829949.845123:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518178688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829949.845131:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.845133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829949.845135:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.845138:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2956 00000100:00080000:0.0:1787829949.845142:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829949.845147:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff8b2e880403c0 x1874672518178304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829949.845148:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829949.845154:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829949.845166:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518178816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829949.845177:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518178304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829949.845181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.845188:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829949.845191:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829949.845198:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2956 00000100:00080000:0.0:1787829949.845200:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829949.845201:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829949.845204:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829949.845210:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518178944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829949.845214:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787829949.845218:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829949.845221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829949.845224:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787829949.845224:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787829949.845231:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2956 00000100:00080000:0.0:1787829949.845236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829949.845236:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518178304:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829949.845243:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829949.845248:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829949.845255:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518179072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829949.845271:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518178432:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829949.845301:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829949.845316:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff8b2e88040000 x1874672518178432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829949.845330:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518178432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829949.845339:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829949.845343:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829949.845347:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518178432:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829949.845354:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518178560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829949.845377:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829949.845390:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff8b2e880430c0 x1874672518178560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829949.845403:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518178560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829949.845411:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829949.845415:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829949.845418:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518178560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829949.845425:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518178688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829949.845445:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829949.845456:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff8b2e88043480 x1874672518178688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829949.845487:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518178688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829949.845497:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829949.845500:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829949.845504:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518178688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829949.845510:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518178816:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829949.845534:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829949.845547:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff8b2e88042940 x1874672518178816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829949.845561:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518178816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829949.845570:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829949.845573:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829949.845577:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518178816:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829949.845584:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518178944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829949.845603:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829949.845614:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff8b2e88040f00 x1874672518178944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829949.845626:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518178944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829949.845635:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829949.845638:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829949.845642:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518178944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829949.845672:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518179072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829949.845695:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829949.845707:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829949/real 1787829949] req@ffff8b2e88041a40 x1874672518179072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829949.845720:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518179072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829974 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829949.845729:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829949.845733:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829949.845736:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518179072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829950.164740:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829950.676738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829950.676878:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829950.868765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829950.868766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829951.188759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829951.700753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829951.700819:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829951.892788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829951.892871:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829952.212806:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829952.724738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829952.724815:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829952.916783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829952.918535:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829953.236765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829953.748761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829953.748820:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829953.940761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829953.940761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829954.260788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829954.772714:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829954.772805:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787829954.964810:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787829954.964820:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829954.964820:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829954.964822:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829954.964828:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829954.964839:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2961 00000100:00080000:0.0:1787829954.964852:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829954.964862:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829954.964900:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518179200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829954.964921:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829954.964924:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829954.964925:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829954.964928:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2961 00000100:00080000:0.0:1787829954.964930:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829954.964933:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829954.964942:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829954.964951:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518179328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829954.964958:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829954.964960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829954.964962:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829954.964966:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2961 00000100:00080000:0.0:1787829954.964969:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829954.964979:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829954.964979:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518179200:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787829954.964987:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829954.964998:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518179456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829954.965006:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829954.965009:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829954.965010:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829954.965013:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2961 00000100:00080000:0.0:1787829954.965016:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829954.965018:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829954.965022:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829954.965033:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518179584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829954.965039:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829954.965041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829954.965042:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829954.965046:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2961 00000100:00080000:0.0:1787829954.965057:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829954.965063:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829954.965068:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829954.965079:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518179712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829954.965089:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829954.965106:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829954.965114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829954.965117:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829954.965124:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2961 00000100:00080000:0.0:1787829954.965133:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829954.965138:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829954.965145:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829954.965169:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518179840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829954.965177:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829954.965180:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829954.965186:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829954.965188:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff8b2e880421c0 x1874672518179200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829954.965195:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2961 00000100:00080000:0.0:1787829954.965203:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829954.965217:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829954.965224:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829954.965234:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518179200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829954.965242:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518179968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829954.965266:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829954.965275:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829954.965279:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518179200:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829954.965311:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518179328:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829954.965336:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829954.965351:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff8b2e88040b40 x1874672518179328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829954.965365:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518179328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829954.965374:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829954.965378:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829954.965381:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518179328:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829954.965388:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518179456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829954.965410:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829954.965439:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff8b2e880412c0 x1874672518179456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829954.965454:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518179456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829954.965463:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829954.965468:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829954.965471:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518179456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829954.965478:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518179584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829954.965498:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829954.965510:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff8b2e88043840 x1874672518179584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829954.965523:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518179584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829954.965532:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829954.965535:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829954.965538:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518179584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829954.965545:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518179712:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829954.965566:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829954.965578:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff8b2e88040780 x1874672518179712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829954.965591:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518179712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829954.965599:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829954.965601:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829954.965603:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518179712:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829954.965608:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518179840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829954.965623:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829954.965630:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff8b2e88042d00 x1874672518179840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829954.965637:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518179840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829954.965666:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829954.965668:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829954.965671:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518179840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829954.965675:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518179968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829954.965689:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829954.965699:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829954/real 1787829954] req@ffff8b2e88042580 x1874672518179968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829954.965709:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518179968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829979 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829954.965717:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829954.965721:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829954.965724:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518179968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829955.284790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829955.796781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829955.796949:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829955.988765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829955.988765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829956.308713:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829956.820758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829956.820816:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829957.012746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829957.012747:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829957.332791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829957.844778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829957.844882:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829958.036764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829958.036774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829958.356777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829958.868790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829958.868852:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829959.060768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829959.060778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829959.380774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829959.892791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829959.892879:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829960.084787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829960.084787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829960.085493:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829960.085499:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829960.085504:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829960.085513:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2966 00000100:00080000:0.0:1787829960.085524:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829960.085530:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829960.085563:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518180096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829960.085593:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829960.085597:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829960.085598:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829960.085602:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2966 00000100:00080000:0.0:1787829960.085606:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829960.085610:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829960.085619:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829960.085628:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518180096:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829960.085635:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518180224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829960.085677:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829960.085681:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829960.085683:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829960.085687:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2966 00000100:00080000:0.0:1787829960.085690:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829960.085694:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829960.085698:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829960.085709:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518180352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829960.085718:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829960.085719:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829960.085725:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829960.085727:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829960.085731:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2966 00000100:00080000:0.0:1787829960.085735:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829960.085738:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829960.085743:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829960.085753:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518180480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829960.085767:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829960/real 1787829960] req@ffff8b2e88041e00 x1874672518180096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829960.085771:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829960.085778:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829960.085782:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829960.085790:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2966 00000100:00080000:0.0:1787829960.085794:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829960.085800:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829960.085801:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518180096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:0.0:1787829960.085804:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829960.085818:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518180608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829960.085819:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829960.085828:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829960.085835:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518180096:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787829960.085843:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829960.085847:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829960.085849:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829960.085854:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2966 00000100:00080000:0.0:1787829960.085857:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829960.085858:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518180224:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829960.085863:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829960.085868:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829960.085880:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518180736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829960.085889:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787829960.085892:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829960.085894:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829960.085904:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829960.085909:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2966 00000100:00100000:3.0:1787829960.085912:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829960/real 1787829960] req@ffff8b2e88042580 x1874672518180224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829960.085914:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829960.085921:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829960.085929:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829960.085935:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518180224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829960.085946:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787829960.085951:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518180864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787829960.085951:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829960.085958:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518180224:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829960.085971:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518180352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829960.085994:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829960.086006:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829960/real 1787829960] req@ffff8b2e88042d00 x1874672518180352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829960.086019:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518180352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829960.086026:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829960.086029:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829960.086032:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518180352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829960.086037:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518180480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829960.086057:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829960.086069:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829960/real 1787829960] req@ffff8b2e88040780 x1874672518180480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829960.086081:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518180480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829960.086088:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829960.086091:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829960.086094:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518180480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829960.086100:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518180608:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829960.086117:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829960.086127:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829960/real 1787829960] req@ffff8b2e88043840 x1874672518180608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829960.086137:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518180608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829960.086145:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829960.086148:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829960.086151:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518180608:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829960.086167:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518180736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829960.086187:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829960.086198:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829960/real 1787829960] req@ffff8b2e880412c0 x1874672518180736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829960.086216:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518180736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829960.086224:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829960.086227:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829960.086230:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518180736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829960.086235:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518180864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829960.086255:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829960.086266:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829960/real 1787829960] req@ffff8b2e88040b40 x1874672518180864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829960.086276:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518180864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829985 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829960.086284:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829960.086287:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829960.086291:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518180864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829960.407458:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829960.916825:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829960.916860:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829961.108731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829961.109447:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829961.428797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829961.940806:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829961.940831:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829962.132755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829962.132755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829962.452805:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829962.964740:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829962.964806:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829963.156768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829963.156811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829963.478509:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829963.988768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829963.988848:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829964.180802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829964.181011:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829964.500778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829965.012865:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829965.013310:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829965.204781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829965.204881:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829965.204887:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829965.204892:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829965.204902:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2971 00000100:00080000:0.0:1787829965.204913:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829965.204919:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829965.204955:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518180992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829965.204965:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829965.204978:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829965.204982:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829965.204984:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829965.204988:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2971 00000100:00080000:0.0:1787829965.204991:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829965.204995:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829965.205004:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829965.205018:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518181120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829965.205027:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829965.205033:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829965.205035:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829965.205039:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2971 00000100:00080000:0.0:1787829965.205044:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829965.205048:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829965.205053:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829965.205063:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518181248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829965.205071:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829965.205074:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829965.205076:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829965.205078:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2971 00000100:00080000:0.0:1787829965.205081:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829965.205084:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829965.205088:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829965.205097:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518181376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829965.205106:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829965.205108:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829965.205110:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829965.205113:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2971 00000100:00080000:0.0:1787829965.205116:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829965.205119:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829965.205124:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829965.205133:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518181504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829965.205141:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829965.205143:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829965.205145:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829965.205148:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2971 00000100:00080000:0.0:1787829965.205151:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829965.205154:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829965.205158:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829965.205167:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518181632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829965.205174:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829965.205177:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829965.205178:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829965.205181:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2971 00000100:00080000:0.0:1787829965.205184:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829965.205188:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829965.205193:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829965.205203:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518181760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829965.205541:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518180992:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787829965.205592:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829965.205630:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829965/real 1787829965] req@ffff8b2e880421c0 x1874672518180992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829965.205645:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518180992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829965.205671:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829965.205674:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829965.205678:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518180992:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829965.205685:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518181120:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787829965.205715:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829965.205727:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829965/real 1787829965] req@ffff8b2e88041a40 x1874672518181120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829965.205738:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518181120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829965.205747:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829965.205750:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829965.205752:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518181120:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829965.205758:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518181248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787829965.205774:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829965.205783:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829965/real 1787829965] req@ffff8b2e88040f00 x1874672518181248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829965.205810:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518181248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829965.205818:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829965.205820:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829965.205824:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518181248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829965.205829:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518181376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787829965.205852:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829965.205870:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829965/real 1787829965] req@ffff8b2e88042940 x1874672518181376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829965.205882:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518181376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829965.205889:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829965.205892:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829965.205895:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518181376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829965.205900:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518181504:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787829965.205915:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829965.205923:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829965/real 1787829965] req@ffff8b2e88043480 x1874672518181504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829965.205933:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518181504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829965.205947:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829965.205950:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829965.205953:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518181504:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829965.205958:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518181632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787829965.205972:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829965.205990:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829965/real 1787829965] req@ffff8b2e880430c0 x1874672518181632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829965.206001:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518181632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829965.206008:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829965.206011:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829965.206014:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518181632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829965.206020:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518181760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787829965.206035:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829965.206043:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829965/real 1787829965] req@ffff8b2e88040000 x1874672518181760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829965.206054:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518181760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829990 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829965.206060:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829965.206063:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829965.206067:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518181760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829965.524762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829966.036752:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829966.036812:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829966.228782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829966.230493:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829966.548785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829967.060730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829967.060810:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829967.252764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829967.252764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829967.572769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787829968.085308:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829968.086037:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829968.276776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829968.276776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829968.597967:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787829969.108807:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787829969.109001:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829969.300808:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829969.301453:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829969.620789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787829970.133466:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829970.134971:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787829970.324763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829970.324767:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829970.324776:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829970.324781:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829970.324879:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2976 00000100:00080000:0.0:1787829970.324890:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829970.324896:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829970.324932:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518181888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829970.325124:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787829970.325186:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829970.325190:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829970.325192:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829970.325197:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2976 00000100:00080000:0.0:1787829970.325201:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829970.325205:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829970.325214:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829970.325230:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518182016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829970.325240:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829970.325243:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829970.325245:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829970.325249:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2976 00000100:00080000:0.0:1787829970.325253:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829970.325256:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829970.325261:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829970.325271:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518182144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829970.325285:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829970.325288:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829970.325290:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829970.325293:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2976 00000100:00080000:0.0:1787829970.325296:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829970.325299:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829970.325302:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829970.325311:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518182272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829970.325454:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518181888:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787829970.325520:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829970.325561:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff8b2e880403c0 x1874672518181888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.325577:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518181888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.325586:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.325589:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829970.325592:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518181888:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829970.325598:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518182016:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787829970.325618:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829970.325643:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff8b2e88040000 x1874672518182016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.325680:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518182016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.325688:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.325691:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829970.325694:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518182016:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829970.325700:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518182144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787829970.325720:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829970.325730:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff8b2e880430c0 x1874672518182144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.325741:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518182144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.325749:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.325752:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829970.325755:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518182144:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829970.325864:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829970.325867:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829970.325869:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829970.325874:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2976 00000100:00080000:0.0:1787829970.325878:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829970.325882:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829970.325888:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829970.325890:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518182272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787829970.325898:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518182400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829970.325908:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787829970.325909:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829970.325911:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829970.325912:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829970.325916:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2976 00000100:00080000:0.0:1787829970.325920:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787829970.325920:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff8b2e88043480 x1874672518182272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829970.325923:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829970.325927:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829970.325930:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518182272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.325938:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.325941:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829970.325945:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518182272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829970.325956:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518182400:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787829970.325974:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829970.325985:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff8b2e88042940 x1874672518182400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.325996:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518182400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.326003:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.326006:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829970.326007:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518182528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829970.326009:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518182400:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829970.326017:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829970.326020:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787829970.326020:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518182528:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829970.326021:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829970.326025:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2976 00000100:00080000:0.0:1787829970.326028:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829970.326031:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829970.326035:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787829970.326037:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829970.326044:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518182656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787829970.326046:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff8b2e88040f00 x1874672518182528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.326057:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518182528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.326064:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.326066:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829970.326069:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518182528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829970.326079:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518182656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787829970.326097:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787829970.326107:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829970/real 1787829970] req@ffff8b2e88041a40 x1874672518182656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829970.326115:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518182656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787829995 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829970.326119:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787829970.326121:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787829970.326123:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518182656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829970.644788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829971.156777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829971.156896:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829971.348812:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829971.348816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829971.668785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829972.180772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829972.180844:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829972.372785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829972.372839:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829972.692814:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829973.204824:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787829973.204837:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829973.396770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829973.396770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829973.716784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829974.228773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829974.228795:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829974.420751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829974.420767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829974.740778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787829975.252801:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829975.252961:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787829975.444791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829975.446178:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829975.446803:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829975.446810:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829975.446814:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829975.446831:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2981 00000100:00080000:1.0:1787829975.446841:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829975.446848:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829975.447307:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518182784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829975.447382:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518182784:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787829975.447439:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829975.447585:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829975.447603:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829975.447605:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829975.447610:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2981 00000100:00080000:1.0:1787829975.447615:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829975.447616:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff8b2e88075e00 x1874672518182784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829975.447624:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787829975.447634:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518182784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829975.447668:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787829975.447681:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829975.447684:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829975.447690:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518182784:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787829975.447797:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518182912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829975.447917:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829975.447922:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829975.447923:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829975.447927:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2981 00000100:00080000:1.0:1787829975.447933:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829975.447937:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829975.447943:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787829975.447949:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518182912:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829975.447957:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518183040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829975.447979:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787829975.447982:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829975.447986:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829975.447990:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829975.447995:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2981 00000100:00080000:1.0:1787829975.447999:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829975.448003:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff8b2e88074780 x1874672518182912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787829975.448007:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829975.448016:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829975.448024:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518182912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787829975.448032:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518183168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787829975.448036:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829975.448043:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829975.448051:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518182912:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829975.448052:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787829975.448064:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518183040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787829975.448082:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829975.448092:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff8b2e88077c00 x1874672518183040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829975.448104:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518183040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829975.448112:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829975.448115:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829975.448118:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518183040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787829975.448123:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518183168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787829975.448137:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829975.448146:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff8b2e88075680 x1874672518183168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829975.448163:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518183168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829975.448171:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829975.448173:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829975.448176:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518183168:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829975.448534:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829975.448536:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829975.448540:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2981 00000100:00080000:1.0:1787829975.448544:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829975.448549:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829975.448554:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829975.448566:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518183296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829975.448582:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829975.448585:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829975.448586:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829975.448590:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2981 00000100:00080000:1.0:1787829975.448592:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829975.448596:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829975.448600:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829975.448611:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518183296:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829975.448612:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518183424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829975.448626:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829975.448629:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829975.448631:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829975.448635:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2981 00000100:00080000:1.0:1787829975.448638:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787829975.448639:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787829975.448644:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829975.448668:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829975.448678:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518183552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829975.448753:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff8b2e88075a40 x1874672518183296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829975.448765:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518183296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829975.448774:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829975.448777:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829975.448781:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518183296:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787829975.448791:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518183424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787829975.448809:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829975.448819:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff8b2e88077480 x1874672518183424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829975.448829:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518183424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829975.448842:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829975.448845:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829975.448848:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518183424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787829975.448853:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518183552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787829975.448866:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829975.448874:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829975/real 1787829975] req@ffff8b2e88076940 x1874672518183552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829975.448884:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518183552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830000 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829975.448891:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829975.448894:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829975.448896:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518183552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829975.764793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829976.276770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829976.276821:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829976.468783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829976.468783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829976.788798:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787829977.300930:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829977.301005:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829977.492781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829977.492781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829977.812803:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829978.324785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829978.324974:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829978.516716:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829978.516724:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829978.836767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787829979.350347:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829979.350371:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829979.540943:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829979.541149:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829979.860819:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787829980.373343:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829980.373415:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829980.564772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829980.565036:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829980.565095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829980.565100:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829980.565104:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829980.565114:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2986 00000100:00080000:0.0:1787829980.565123:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829980.565129:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829980.565165:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518183680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829980.565230:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518183680:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787829980.565295:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829980.565325:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829980.565329:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829980.565331:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829980.565334:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2986 00000100:00080000:0.0:1787829980.565338:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829980.565342:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829980.565352:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829980.565367:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518183808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829980.565422:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff8b2e880421c0 x1874672518183680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829980.565438:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518183680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829980.565448:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829980.565452:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829980.565455:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518183680:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829980.565475:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518183808:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787829980.565496:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829980.565510:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff8b2e88040b40 x1874672518183808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829980.565522:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518183808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829980.565531:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829980.565534:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829980.565536:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518183808:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829980.565616:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829980.565620:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829980.565622:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829980.565626:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2986 00000100:00080000:0.0:1787829980.565629:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829980.565633:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829980.565638:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829980.565672:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518183936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829980.565714:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518183936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787829980.565737:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829980.565757:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829980.565760:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829980.565761:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829980.565765:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2986 00000100:00080000:0.0:1787829980.565769:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829980.565772:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829980.565777:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829980.565789:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518184064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829980.565827:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff8b2e880412c0 x1874672518183936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829980.565845:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518183936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829980.565852:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829980.565855:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829980.565858:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518183936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787829980.565869:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518184064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787829980.565890:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829980.565900:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff8b2e88043840 x1874672518184064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829980.565910:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518184064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829980.565918:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829980.565921:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829980.565923:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518184064:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829980.565998:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829980.566002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829980.566004:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829980.566009:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2986 00000100:00080000:0.0:1787829980.566013:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829980.566016:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829980.566021:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829980.566033:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518184192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829980.566073:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518184192:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787829980.566094:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829980.566112:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829980.566116:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829980.566117:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829980.566121:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2986 00000100:00080000:0.0:1787829980.566124:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829980.566128:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829980.566133:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829980.566144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518184320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829980.566180:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff8b2e88040780 x1874672518184192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829980.566191:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518184192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829980.566204:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829980.566207:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829980.566210:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518184192:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787829980.566221:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518184320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787829980.566240:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829980.566262:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff8b2e88042d00 x1874672518184320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829980.566274:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518184320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829980.566281:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829980.566284:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829980.566287:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518184320:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787829980.566361:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829980.566365:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829980.566367:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829980.566371:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2986 00000100:00080000:0.0:1787829980.566375:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829980.566378:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829980.566383:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829980.566394:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518184448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829980.566433:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518184448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787829980.566454:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829980.566507:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829980/real 1787829980] req@ffff8b2e88042580 x1874672518184448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829980.566519:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518184448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830005 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829980.566528:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829980.566531:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829980.566535:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518184448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829980.884766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787829981.397044:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829981.397088:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787829981.588719:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829981.588744:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829981.910192:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829982.420714:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787829982.420976:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829982.613167:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829982.613660:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829982.932870:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829983.444776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829983.444861:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829983.636766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787829983.636772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829983.956810:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829984.468721:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829984.469251:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829984.660799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787829984.660807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829984.980789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829985.492771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829985.492866:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829985.684794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829985.684901:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829985.684907:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829985.684910:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829985.684920:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2992 00000100:00080000:0.0:1787829985.684930:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829985.684937:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829985.684976:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518184576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829985.685001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829985.685004:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829985.685006:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829985.685009:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2992 00000100:00080000:0.0:1787829985.685012:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829985.685015:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829985.685024:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829985.685035:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518184704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829985.685048:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829985.685051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829985.685052:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829985.685055:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2992 00000100:00080000:0.0:1787829985.685058:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829985.685061:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829985.685064:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829985.685072:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518184832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829985.685079:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829985.685082:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829985.685083:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829985.685085:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2992 00000100:00080000:0.0:1787829985.685088:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829985.685090:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829985.685093:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829985.685100:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518184960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829985.685107:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829985.685109:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829985.685111:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829985.685114:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2992 00000100:00080000:0.0:1787829985.685117:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829985.685120:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829985.685123:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829985.685129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518185088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829985.685136:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829985.685138:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829985.685139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829985.685142:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2992 00000100:00080000:0.0:1787829985.685145:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829985.685147:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829985.685150:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829985.685157:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518185216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829985.685163:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829985.685165:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829985.685166:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829985.685169:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2992 00000100:00080000:0.0:1787829985.685173:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829985.685176:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829985.685179:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787829985.685186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518185344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829985.686317:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518184576:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829985.686337:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000800:00000100:2.0:1787829985.686383:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829985.686424:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff8b2e88041e00 x1874672518184576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829985.686438:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518184576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829985.686447:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829985.686450:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829985.686453:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518184576:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787829985.686460:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518184704:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787829985.686633:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829985.686663:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff8b2e88042580 x1874672518184704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829985.686674:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518184704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829985.686682:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829985.686684:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829985.686687:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518184704:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787829985.686693:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518184832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787829985.686712:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829985.686755:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff8b2e88042d00 x1874672518184832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829985.686766:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518184832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829985.686773:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829985.686775:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829985.686778:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518184832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787829985.686784:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518184960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787829985.686802:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829985.686812:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff8b2e88040780 x1874672518184960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829985.686821:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518184960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829985.686828:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829985.686830:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829985.686833:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518184960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787829985.686838:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518185088:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787829985.686853:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829985.686862:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff8b2e88043840 x1874672518185088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829985.686871:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518185088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829985.686878:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829985.686880:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829985.686883:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518185088:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787829985.686888:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518185216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787829985.686903:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829985.686911:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff8b2e880412c0 x1874672518185216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829985.687094:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518185216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829985.687103:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829985.687106:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829985.687109:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518185216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787829985.687115:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518185344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787829985.687135:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787829985.687145:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829985/real 1787829985] req@ffff8b2e88040b40 x1874672518185344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787829985.687155:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518185344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830010 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787829985.687161:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787829985.687164:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787829985.687167:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518185344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829986.004745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829986.516903:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829986.516977:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829986.708890:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829986.709009:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829987.028824:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787829987.541524:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829987.542270:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787829987.732792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829987.732896:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829988.052912:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829988.564785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829988.565379:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829988.756775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829988.756775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829989.078249:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829989.588778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829989.588861:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829989.780770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829989.780770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829990.100788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829990.612796:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829990.612859:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829990.804767:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829990.804835:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787829990.805254:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829990.805259:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787829990.805262:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829990.805270:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 2997 00000100:00080000:1.0:1787829990.805281:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829990.805288:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787829990.805321:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518185472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829990.805352:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829990.805355:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829990.805357:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829990.805361:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2997 00000100:00080000:1.0:1787829990.805365:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829990.805368:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829990.805377:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829990.805391:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518185600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829990.805391:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518185472:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787829990.805400:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829990.805403:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829990.805404:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829990.805407:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2997 00000100:00080000:1.0:1787829990.805410:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829990.805412:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829990.805416:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787829990.805423:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518185728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829990.805430:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829990.805433:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829990.805434:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829990.805437:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 2997 00000100:00080000:1.0:1787829990.805439:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829990.805442:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829990.805446:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829990.805453:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518185856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829990.805460:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829990.805463:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787829990.805464:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829990.805468:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2997 00000100:00080000:1.0:1787829990.805471:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829990.805473:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829990.805477:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787829990.805477:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518185600:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787829990.805487:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518185984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787829990.805496:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829990.805499:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787829990.805500:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787829990.805502:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518185728:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829990.805504:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2997 00000100:00080000:1.0:1787829990.805506:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787829990.805509:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829990.805513:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829990.805523:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518186112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829990.805523:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518185856:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787829990.805532:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787829990.805535:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787829990.805536:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787829990.805539:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 2997 00000100:00080000:1.0:1787829990.805542:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787829990.805544:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518185984:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787829990.805545:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787829990.805550:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787829990.805558:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518186240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787829990.805562:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518186112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787829990.805582:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518186240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829991.125191:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829991.636780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829991.636854:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829991.828777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829991.828807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829992.149324:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787829992.660880:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787829992.660924:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787829992.852796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829992.852797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787829993.172782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829993.684776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829993.684842:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829993.876744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:2.0:1787829993.876989:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787829993.877033:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787829993.877055:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:1.0:1787829993.877066:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000800:00000100:2.0:1787829993.877096:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829993.877106:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829993.877116:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829993.877126:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829993.877127:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829993] req@ffff8b2e88076940 x1874672518185600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:2.0:1787829993.877135:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787829993.877146:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787829993.877177:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518185600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829993.877189:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829993.877195:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829993.877199:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518185600:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829993.877206:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829993] req@ffff8b2e88077480 x1874672518185728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829993.877219:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518185728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829993.877226:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829993.877229:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829993.877232:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518185728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829993.877238:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829993] req@ffff8b2e88075a40 x1874672518185856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829993.877249:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518185856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829993.877257:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829993.877260:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829993.877263:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518185856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829993.877269:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829993] req@ffff8b2e88074f00 x1874672518185472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829993.877282:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518185472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829993.877289:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829993.877292:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829993.877295:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518185472:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787829993.877320:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829993] req@ffff8b2e88074780 x1874672518186240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829993.877332:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518186240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829993.877353:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829993.877356:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829993.877359:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518186240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829993.877365:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829993] req@ffff8b2e88075680 x1874672518185984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829993.877377:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518185984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829993.877385:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829993.877388:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829993.877391:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518185984:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829993.877397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829990/real 1787829993] req@ffff8b2e88077c00 x1874672518186112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829993.877408:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518186112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830015 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829993.877416:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829993.877418:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829993.877421:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518186112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829994.197065:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829994.708762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829994.708827:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829994.900778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829994.901152:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829995.220764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829995.732720:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829995.732817:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787829995.924730:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829995.924740:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787829995.924795:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.924802:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787829995.924805:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829995.924813:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3002 00000100:00080000:0.0:1787829995.924823:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829995.924829:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787829995.924866:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518186368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829995.924890:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.924894:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829995.924895:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829995.924899:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3002 00000100:00080000:0.0:1787829995.924903:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829995.924907:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829995.924918:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829995.924922:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518186368:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829995.924935:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518186496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829995.924946:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.924950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829995.924952:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829995.924956:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3002 00000100:00080000:0.0:1787829995.924960:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829995.924964:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829995.924969:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787829995.924981:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518186624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787829995.924983:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787829995.924994:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.924998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829995.925000:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829995.925004:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3002 00000100:00080000:0.0:1787829995.925008:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829995.925011:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829995.925016:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787829995.925027:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff8b2e880421c0 x1874672518186368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787829995.925029:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518186752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829995.925056:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.925059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787829995.925061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787829995.925064:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518186368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829995.925068:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3002 00000100:00080000:0.0:1787829995.925074:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829995.925080:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787829995.925082:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787829995.925086:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829995.925089:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829995.925095:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518186368:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787829995.925100:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518186880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787829995.925110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.925113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787829995.925115:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829995.925121:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3002 00000100:00100000:3.0:1787829995.925121:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518186496:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787829995.925127:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787829995.925132:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829995.925155:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787829995.925160:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787829995.925170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518187008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829995.925178:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff8b2e88041a40 x1874672518186496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829995.925185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787829995.925193:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787829995.925196:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787829995.925206:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3002 00000100:00080000:0.0:1787829995.925211:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787829995.925215:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518186496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787829995.925218:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787829995.925228:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787829995.925231:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829995.925237:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787829995.925242:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518187136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787829995.925243:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518186496:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829995.925255:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518186624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829995.925278:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829995.925289:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff8b2e88040f00 x1874672518186624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829995.925301:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518186624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829995.925308:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829995.925311:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829995.925314:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518186624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829995.925320:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518186752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829995.925337:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829995.925346:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff8b2e88042940 x1874672518186752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829995.925357:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518186752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829995.925371:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829995.925374:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829995.925376:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518186752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829995.925382:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518186880:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787829995.925399:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829995.925410:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff8b2e88043480 x1874672518186880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829995.925420:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518186880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829995.925428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829995.925431:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829995.925434:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518186880:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787829995.925450:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518187008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829995.925468:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829995.925479:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff8b2e880430c0 x1874672518187008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829995.925491:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518187008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829995.925498:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829995.925500:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829995.925504:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518187008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787829995.925509:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518187136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787829995.925525:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787829995.925536:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787829995/real 1787829995] req@ffff8b2e88040000 x1874672518187136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787829995.925547:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518187136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830020 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787829995.925556:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787829995.925559:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787829995.925561:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518187136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787829996.244711:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829996.756754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829996.756814:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829996.948777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829996.948777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829997.268755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829997.780723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829997.780780:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829997.972778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829997.972778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829998.293696:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829998.804749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829998.804823:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787829998.996748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787829998.996748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787829999.316755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787829999.828729:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787829999.828790:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830000.020763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830000.020763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830000.340730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830000.852756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830000.852825:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830001.044739:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830001.044841:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830001.044857:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830001.044860:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830001.044869:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3007 00000100:00080000:1.0:1787830001.044879:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830001.044885:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830001.044922:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518187264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830001.044950:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830001.044954:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830001.044961:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830001.044966:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3007 00000100:00080000:1.0:1787830001.044970:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830001.044975:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830001.044985:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830001.044995:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518187264:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830001.045001:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518187392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830001.045009:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830001.045011:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830001.045012:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830001.045015:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3007 00000100:00080000:1.0:1787830001.045017:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830001.045020:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830001.045025:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830001.045035:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518187520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830001.045042:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830001.045044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830001.045045:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830001.045047:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3007 00000800:00000100:2.0:1787830001.045066:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830001.045089:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830001.045138:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830001.045144:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830001.045149:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830001.045161:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518187648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830001.045171:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830001.045176:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:2.0:1787830001.045176:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830001/real 1787830001] req@ffff8b2e88075e00 x1874672518187264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830001.045180:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830001.045189:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3007 00000100:00080000:1.0:1787830001.045196:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830001.045201:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830001.045204:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518187264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787830001.045207:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787830001.045220:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787830001.045221:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518187776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830001.045229:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787830001.045236:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830001.045237:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518187264:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830001.045243:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830001.045247:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830001.045253:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3007 00000100:00080000:1.0:1787830001.045256:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830001.045260:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830001.045264:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830001.045271:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518187392:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830001.045279:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518187904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830001.045287:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830001.045289:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830001.045291:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830001.045293:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3007 00000100:00080000:1.0:1787830001.045297:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787830001.045299:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830001.045302:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830001.045307:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830001.045318:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830001/real 1787830001] req@ffff8b2e880761c0 x1874672518187392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830001.045319:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518188032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830001.045354:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518187392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830001.045364:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830001.045368:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830001.045371:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518187392:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830001.045377:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518187520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830001.045393:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830001.045402:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830001/real 1787830001] req@ffff8b2e88074000 x1874672518187520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830001.045413:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518187520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830001.045421:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830001.045424:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830001.045427:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518187520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830001.045432:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518187648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830001.045445:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830001.045453:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830001/real 1787830001] req@ffff8b2e88074b40 x1874672518187648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830001.045470:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518187648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830001.045477:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830001.045479:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830001.045483:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518187648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830001.045487:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518187776:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830001.045501:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830001.045509:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830001/real 1787830001] req@ffff8b2e880770c0 x1874672518187776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830001.045519:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518187776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830001.045528:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830001.045530:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830001.045533:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518187776:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830001.045547:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518187904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830001.045560:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830001.045569:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830001/real 1787830001] req@ffff8b2e88077840 x1874672518187904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830001.045579:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518187904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830001.045594:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830001.045597:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830001.045599:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518187904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830001.045604:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518188032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830001.045629:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830001.045638:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830001/real 1787830001] req@ffff8b2e88076d00 x1874672518188032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830001.045671:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518188032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830026 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830001.045679:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830001.045682:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830001.045685:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518188032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830001.364992:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830001.876815:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830001.876915:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830002.068799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830002.068799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830002.388819:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830002.900776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830002.900967:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830003.092830:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830003.092836:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830003.412782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830003.924847:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830003.925943:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830004.116763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830004.117091:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830004.436815:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830004.948857:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830004.949400:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830005.140791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830005.141147:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830005.462883:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830005.972782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830005.972999:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830006.164790:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830006.164794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830006.164802:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830006.164808:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830006.164817:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3012 00000100:00080000:1.0:1787830006.164828:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830006.164835:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830006.164876:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518188160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830006.164901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830006.164905:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830006.164907:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830006.164911:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3012 00000100:00080000:1.0:1787830006.164914:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830006.164918:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830006.164927:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830006.164942:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518188288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830006.164951:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830006.164954:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830006.164956:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830006.164960:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3012 00000100:00080000:1.0:1787830006.164963:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830006.164967:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830006.164973:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830006.164983:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518188416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830006.164993:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830006.164998:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830006.165000:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830006.165000:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518188160:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830006.165008:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3012 00000100:00080000:1.0:1787830006.165013:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830006.165016:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830006.165021:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830006.165033:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518188544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830006.165041:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830006.165045:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830006.165046:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830006.165050:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3012 00000100:00080000:1.0:1787830006.165056:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830006.165060:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830006.165065:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787830006.165077:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830006.165084:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830006.165125:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830006/real 1787830006] req@ffff8b2e88076580 x1874672518188160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830006.165129:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518188672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830006.165148:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518188160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830006.165168:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830006.165175:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830006.165176:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787830006.165182:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787830006.165183:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518188160:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830006.165187:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830006.165194:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3012 00000100:00100000:2.0:1787830006.165196:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518188288:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830006.165199:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830006.165213:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830006.165219:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787830006.165225:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830006.165233:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518188800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830006.165241:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830006/real 1787830006] req@ffff8b2e88076d00 x1874672518188288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830006.165246:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830006.165254:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830006.165258:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830006.165265:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3012 00000100:00100000:2.0:1787830006.165265:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518188288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830006.165272:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830006.165278:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787830006.165284:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787830006.165286:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787830006.165292:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830006.165298:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518188288:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830006.165302:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518188928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830006.165326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518188416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830006.165349:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830006.165360:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830006/real 1787830006] req@ffff8b2e88077840 x1874672518188416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830006.165371:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518188416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830006.165385:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830006.165388:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830006.165391:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518188416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830006.165397:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518188544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830006.165413:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830006.165421:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830006/real 1787830006] req@ffff8b2e880770c0 x1874672518188544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830006.165431:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518188544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830006.165439:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830006.165442:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830006.165444:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518188544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830006.165450:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518188672:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830006.165466:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830006.165476:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830006/real 1787830006] req@ffff8b2e88074b40 x1874672518188672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830006.165487:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518188672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830006.165501:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830006.165505:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830006.165508:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518188672:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830006.165514:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518188800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830006.165530:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830006.165541:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830006/real 1787830006] req@ffff8b2e88074000 x1874672518188800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830006.165551:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518188800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830006.165559:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830006.165562:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830006.165565:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518188800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830006.165570:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518188928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830006.165588:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830006.165598:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830006/real 1787830006] req@ffff8b2e880761c0 x1874672518188928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830006.165609:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518188928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830031 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830006.165616:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830006.165619:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830006.165623:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518188928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830006.484773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830006.996775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830006.996894:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830007.188838:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830007.189107:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830007.508955:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830008.020731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830008.020949:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830008.213426:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830008.213451:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830008.532723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830009.044766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830009.044861:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830009.236765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830009.236767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830009.556774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830010.068757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830010.068865:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830010.260793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830010.260820:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830010.580829:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830011.092796:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830011.093212:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830011.285111:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830011.285119:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.285127:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830011.285130:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.285138:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3017 00000100:00080000:0.0:1787830011.285147:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.285152:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830011.285186:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518189056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830011.285219:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.285221:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830011.285223:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.285225:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3017 00000100:00080000:0.0:1787830011.285228:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.285231:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830011.285239:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830011.285249:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518189184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830011.285256:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.285259:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830011.285261:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.285263:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3017 00000100:00080000:0.0:1787830011.285265:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.285267:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830011.285271:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830011.285273:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518189056:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830011.285279:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518189312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830011.285285:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.285287:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830011.285288:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.285289:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3017 00000100:00080000:0.0:1787830011.285293:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.285294:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830011.285297:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830011.285311:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518189440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830011.285314:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830011.285322:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.285326:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830011.285328:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.285332:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3017 00000100:00080000:0.0:1787830011.285336:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787830011.285340:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830011.285341:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830011.285346:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830011.285353:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518189568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830011.285360:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830011.285362:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830011.285363:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.285365:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3017 00000100:00080000:0.0:1787830011.285367:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.285369:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830011.285371:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830011.285377:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518189696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830011.285388:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830011.285388:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff8b2e880403c0 x1874672518189056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830011.285395:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830011.285397:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830011.285403:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3017 00000100:00080000:0.0:1787830011.285407:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830011.285410:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830011.285413:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830011.285419:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518189056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830011.285421:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518189824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830011.285688:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830011.285692:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830011.285696:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518189056:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830011.285703:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518189184:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830011.285736:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830011.285749:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff8b2e88040000 x1874672518189184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830011.285759:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518189184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830011.285767:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830011.285770:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830011.285782:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518189184:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830011.285802:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518189312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830011.285821:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830011.285830:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff8b2e880430c0 x1874672518189312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830011.285840:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518189312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830011.285847:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830011.285850:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830011.285852:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518189312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830011.285857:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518189440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830011.285870:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830011.285878:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff8b2e88043480 x1874672518189440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830011.285887:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518189440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830011.285894:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830011.285897:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830011.285900:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518189440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830011.285914:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518189568:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830011.285930:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830011.285939:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff8b2e88042940 x1874672518189568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830011.285955:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518189568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830011.285962:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830011.285964:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830011.285967:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518189568:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830011.285972:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518189696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830011.285986:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830011.285997:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff8b2e88040f00 x1874672518189696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830011.286007:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518189696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830011.286013:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830011.286016:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830011.286019:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518189696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830011.286024:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518189824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830011.286051:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830011.286061:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830011/real 1787830011] req@ffff8b2e88041a40 x1874672518189824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830011.286071:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518189824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830036 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830011.286082:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830011.286084:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830011.286087:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518189824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830011.604761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830012.117523:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830012.119369:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830012.308838:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830012.308878:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830012.628785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830013.140766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830013.141770:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830013.332744:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830013.332746:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830013.652759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830014.164765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830014.164840:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830014.356932:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830014.356933:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830014.676865:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830015.188771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830015.188890:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830015.380766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830015.380766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830015.700751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830016.212779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830016.212938:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830016.404713:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830016.404727:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787830016.404765:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.404772:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830016.404776:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.404784:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3022 00000100:00080000:2.0:1787830016.404794:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.404800:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830016.404835:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518189952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830016.404865:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.404868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830016.404870:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.404874:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3022 00000100:00080000:2.0:1787830016.404877:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.404880:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830016.404889:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830016.404901:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518190080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830016.404910:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.404913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830016.404916:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.404920:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3022 00000100:00080000:2.0:1787830016.404923:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.404927:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830016.404931:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830016.404933:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518189952:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830016.404944:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518190208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830016.404955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.404958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830016.404960:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.404963:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3022 00000100:00080000:2.0:1787830016.404967:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.404970:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830016.404974:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830016.404984:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518190336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830016.404993:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787830016.405010:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787830016.405011:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830016.405015:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830016.405020:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3022 00000100:00080000:2.0:1787830016.405024:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.405029:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830016.405033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830016.405043:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518190464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830016.405053:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830016.405056:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830016.405058:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830016.405061:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff8b2e8801da40 x1874672518189952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830016.405065:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3022 00000100:00080000:2.0:1787830016.405071:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830016.405077:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830016.405084:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830016.405099:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518190592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830016.405099:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518189952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830016.405128:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830016.405129:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830016.405133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830016.405137:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830016.405137:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830016.405143:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518189952:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787830016.405145:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3022 00000100:00080000:2.0:1787830016.405152:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830016.405158:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518190080:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787830016.405159:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830016.405167:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830016.405180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518190720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830016.405192:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830016.405211:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff8b2e8801fc00 x1874672518190080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830016.405225:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518190080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830016.405233:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830016.405235:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830016.405238:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518190080:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830016.405258:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518190208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830016.405275:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830016.405285:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff8b2e8801f480 x1874672518190208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830016.405297:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518190208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830016.405322:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830016.405325:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830016.405328:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518190208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830016.405333:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518190336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830016.405349:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830016.405359:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff8b2e8801cf00 x1874672518190336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830016.405368:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518190336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830016.405374:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830016.405376:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830016.405379:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518190336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830016.405384:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518190464:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830016.405399:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830016.405409:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff8b2e8801c000 x1874672518190464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830016.405418:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518190464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830016.405426:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830016.405429:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830016.405432:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518190464:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830016.405447:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518190592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830016.405465:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830016.405476:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff8b2e8801e580 x1874672518190592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830016.405486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518190592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830016.405493:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830016.405495:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830016.405498:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518190592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830016.405513:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518190720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830016.405529:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830016.405540:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830016/real 1787830016] req@ffff8b2e8801e940 x1874672518190720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830016.405549:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518190720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830041 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830016.405556:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830016.405558:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830016.405560:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518190720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830016.724791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830017.236948:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830017.237134:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830017.428790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830017.429103:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830017.748788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830018.260761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830018.260904:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830018.452776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830018.452776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830018.772804:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830019.284748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830019.284875:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830019.476852:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830019.476930:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830019.797227:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830020.308778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830020.309206:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830020.501006:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830020.501480:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830020.822215:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830021.332860:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830021.332865:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830021.524762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830021.524844:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830021.524850:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830021.524854:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830021.524864:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3027 00000100:00100000:1.0:1787830021.524864:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830021.524892:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830021.524900:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830021.524941:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518190848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830021.525034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830021.525038:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830021.525040:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830021.525045:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3027 00000100:00080000:0.0:1787830021.525049:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830021.525053:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830021.525063:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830021.525079:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518190976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830021.525081:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518190848:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787830021.525097:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830021.525101:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830021.525103:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830021.525107:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3027 00000100:00080000:0.0:1787830021.525111:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830021.525115:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830021.525121:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830021.525134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518191104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830021.525143:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787830021.525146:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830021.525148:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830021.525152:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830021.525156:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3027 00000100:00080000:0.0:1787830021.525162:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830021.525166:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830021.525170:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830021.525182:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518191232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830021.525190:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830021.525194:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830021.525196:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830021.525200:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3027 00000100:00080000:0.0:1787830021.525204:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830021.525207:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff8b2e880421c0 x1874672518190848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830021.525211:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830021.525216:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830021.525227:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518191360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830021.525237:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830021.525240:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830021.525241:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830021.525245:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3027 00000100:00080000:0.0:1787830021.525248:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830021.525252:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830021.525255:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830021.525264:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518191488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830021.525272:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830021.525275:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830021.525277:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830021.525280:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3027 00000100:00080000:0.0:1787830021.525283:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830021.525286:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830021.525290:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830021.525330:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518190848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830021.525340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830021.525344:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830021.525349:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518190848:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830021.525369:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518191616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830021.525373:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518190976:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830021.525406:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830021.525490:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff8b2e88040b40 x1874672518190976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830021.525505:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518190976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830021.525513:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830021.525516:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830021.525521:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518190976:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830021.525527:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518191104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830021.525548:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830021.525560:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff8b2e880412c0 x1874672518191104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830021.525572:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518191104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830021.525580:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830021.525583:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830021.525586:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518191104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830021.525592:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518191232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830021.525610:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830021.525621:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff8b2e88043840 x1874672518191232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830021.525632:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518191232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830021.525640:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830021.525643:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830021.525667:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518191232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830021.525674:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518191360:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830021.525694:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830021.525706:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff8b2e88040780 x1874672518191360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830021.525717:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518191360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830021.525725:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830021.525728:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830021.525731:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518191360:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830021.525737:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518191488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830021.525762:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830021.525774:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff8b2e88042d00 x1874672518191488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830021.525785:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518191488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830021.525793:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830021.525795:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830021.525798:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518191488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830021.525816:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518191616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830021.525834:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830021.525845:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830021/real 1787830021] req@ffff8b2e88042580 x1874672518191616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830021.525856:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518191616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830046 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830021.525864:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830021.525867:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830021.525870:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518191616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830021.844770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830022.357579:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830022.358158:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830022.548776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830022.548798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830022.869097:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830023.380765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830023.380937:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830023.572764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830023.574361:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830023.892786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830024.404777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830024.404872:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830024.597138:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830024.597205:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830024.916882:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830025.428767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830025.428835:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830025.620852:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830025.620915:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830025.940988:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830026.452904:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830026.452949:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830026.645402:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.645412:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830026.645416:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.645425:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3033 00000100:00080000:3.0:1787830026.645436:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.645443:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830026.645491:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518191744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830026.645522:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.645526:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830026.645528:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.645532:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3033 00000100:00080000:3.0:1787830026.645535:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.645539:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830026.645548:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830026.645560:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518191872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830026.645570:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.645573:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830026.645575:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.645578:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3033 00000100:00080000:3.0:1787830026.645580:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.645584:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830026.645588:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830026.645598:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518192000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830026.645607:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.645610:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830026.645612:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.645615:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3033 00000100:00080000:3.0:1787830026.645617:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.645620:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830026.645624:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830026.645633:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518192128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830026.645641:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.645663:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830026.645665:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.645669:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3033 00000100:00080000:3.0:1787830026.645672:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.645676:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830026.645680:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830026.645692:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518192256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830026.645701:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.645704:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830026.645706:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.645710:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3033 00000100:00080000:3.0:1787830026.645713:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.645717:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830026.645722:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830026.645730:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518191744:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830026.645735:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518192384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830026.645746:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830026.645750:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830026.645752:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830026.645755:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3033 00000100:00080000:3.0:1787830026.645758:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830026.645762:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830026.645767:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830026.645777:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518192512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830026.645794:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830026.645840:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff8b2e88038f00 x1874672518191744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830026.645856:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830026.645860:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518191744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.645873:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.645879:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830026.645883:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518191744:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830026.645890:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518191872:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830026.645911:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830026.645936:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff8b2e8803a1c0 x1874672518191872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.645950:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518191872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.645959:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.645962:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830026.645965:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518191872:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830026.645972:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518192000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830026.645995:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830026.646005:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff8b2e8803a580 x1874672518192000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.646017:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518192000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.646024:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.646028:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830026.646031:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518192000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830026.646036:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518192128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830026.646054:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830026.646064:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff8b2e8803b840 x1874672518192128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.646075:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518192128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.646083:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.646086:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830026.646089:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518192128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830026.646094:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518192256:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830026.646109:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830026.646119:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff8b2e8803b0c0 x1874672518192256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.646135:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518192256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.646143:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.646146:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830026.646149:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518192256:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830026.646174:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518192384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830026.646192:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830026.646201:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff8b2e880383c0 x1874672518192384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.646211:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518192384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.646219:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.646222:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830026.646224:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518192384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830026.646229:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518192512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830026.646245:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830026.646254:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830026/real 1787830026] req@ffff8b2e88039680 x1874672518192512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830026.646264:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518192512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830051 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830026.646277:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830026.646280:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830026.646283:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518192512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830026.648105:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830026.964770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830027.476786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830027.477669:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830027.668776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830027.668990:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830027.989485:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830028.500781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830028.501185:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830028.692793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830028.693293:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830029.012790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830029.524761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830029.524888:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830029.716982:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830029.717778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830030.036726:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830030.548810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830030.549506:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830030.740739:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830030.740775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830031.060788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830031.572738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830031.572996:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830031.764786:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.764797:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:1.0:1787830031.764800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830031.764802:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.764814:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3038 00000100:00100000:2.0:1787830031.764829:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830031.764830:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.764841:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830031.764900:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518192640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.764927:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.764931:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830031.764933:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.764937:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3038 00000100:00080000:3.0:1787830031.764940:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.764945:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830031.764954:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830031.764967:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518192768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.764976:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.764980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830031.764981:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.764985:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3038 00000100:00080000:3.0:1787830031.764988:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.764991:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830031.764995:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830031.765005:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518192896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.765013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.765016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830031.765018:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.765021:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3038 00000100:00080000:3.0:1787830031.765024:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.765028:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830031.765033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830031.765052:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518192640:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830031.765060:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518193024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.765070:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.765074:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830031.765076:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.765081:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3038 00000100:00080000:3.0:1787830031.765085:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.765089:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830031.765094:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830031.765107:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518193152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830031.765113:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830031.765122:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.765125:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830031.765127:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.765132:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3038 00000100:00080000:3.0:1787830031.765137:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830031.765149:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830031.765155:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830031.765161:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830031] req@ffff8b2e88039a40 x1874672518192640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830031.765168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518193280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830031.765181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830031.765186:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830031.765188:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830031.765191:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3038 00000100:00080000:3.0:1787830031.765195:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830031.765196:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518192640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830031.765200:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830031.765209:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787830031.765215:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830031.765222:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830031.765223:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518193408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830031.765227:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518192640:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830031.765235:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518192768:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830031.765269:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830031.765281:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830031] req@ffff8b2e88039e00 x1874672518192768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830031.765365:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518192768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830031.765374:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830031.765378:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830031.765381:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518192768:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830031.765388:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518192896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830031.765410:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830031.765422:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830031] req@ffff8b2e8803b480 x1874672518192896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830031.765445:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518192896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830031.765454:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830031.765457:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830031.765461:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518192896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830031.765485:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518193024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830031.765505:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830031.765515:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830031] req@ffff8b2e880392c0 x1874672518193024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830031.765535:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518193024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830031.765543:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830031.765547:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830031.765550:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518193024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830031.765555:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518193152:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830031.765572:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830031.765582:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830031] req@ffff8b2e88039680 x1874672518193152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830031.765593:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518193152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830031.765600:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830031.765603:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830031.765606:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518193152:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830031.765611:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518193280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830031.765628:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830031.765638:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830031] req@ffff8b2e880383c0 x1874672518193280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830031.765676:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518193280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830031.765684:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830031.765688:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830031.765691:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518193280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830031.765697:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518193408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830031.765717:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830031.765727:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830031/real 1787830031] req@ffff8b2e8803b0c0 x1874672518193408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830031.765738:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518193408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830056 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830031.765751:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830031.765754:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830031.765757:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518193408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830032.084795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830032.596839:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830032.596852:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830032.788786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830032.788786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830033.108835:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830033.620770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830033.620844:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830033.812703:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830033.812742:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830034.134014:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830034.644888:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830034.644936:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830034.836799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830034.837804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830035.157066:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830035.668785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830035.669012:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830035.860786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830035.860786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830036.180791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830036.692731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830036.692906:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830036.884792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830036.884795:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830036.884799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830036.884806:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830036.884811:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830036.884820:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3043 00000100:00080000:1.0:1787830036.884830:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830036.884837:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830036.884884:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518193536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830036.884908:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830036.884912:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830036.884914:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830036.884918:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3043 00000100:00080000:1.0:1787830036.884922:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830036.884927:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830036.884937:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830036.884945:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518193536:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830036.884952:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518193664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830036.884964:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830036.884968:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830036.884970:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830036.884973:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3043 00000100:00080000:1.0:1787830036.884977:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830036.884992:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830036.884998:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:0.0:1787830036.885009:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830036.885012:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518193792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830036.885022:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830036.885026:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830036.885028:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830036.885032:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3043 00000100:00080000:1.0:1787830036.885035:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830036.885039:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830036.885044:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830036.885056:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518193920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830036.885059:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff8b2e88075e00 x1874672518193536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830036.885074:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830036.885079:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830036.885082:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830036.885088:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518193536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830036.885088:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3043 00000100:00080000:1.0:1787830036.885095:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830036.885102:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787830036.885106:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787830036.885109:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787830036.885114:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830036.885119:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518193536:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830036.885141:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518193664:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830036.885168:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830036.885180:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff8b2e88077c00 x1874672518193664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830036.885185:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518194048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830036.885198:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518193664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.885223:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830036.885226:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830036.885229:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787830036.885232:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830036.885235:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830036.885236:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518193664:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830036.885243:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3043 00000100:00100000:0.0:1787830036.885248:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518193792:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787830036.885249:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830036.885257:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830036.885264:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830036.885276:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830036.885287:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff8b2e88075680 x1874672518193792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.885297:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518193792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.885304:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.885307:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830036.885310:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518193792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830036.885316:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518193920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830036.885333:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830036.885343:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff8b2e88074780 x1874672518193920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.885353:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518193920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.885360:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.885363:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830036.885366:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518193920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830036.885381:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518194048:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830036.885409:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830036.885421:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff8b2e88074f00 x1874672518194048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.885444:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518194048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.885452:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.885455:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830036.885458:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518194048:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830036.885533:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518194176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830036.885549:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830036.885552:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830036.885554:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830036.885558:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3043 00000100:00080000:1.0:1787830036.885562:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830036.885566:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830036.885571:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830036.885582:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518194304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830036.885701:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518194176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830036.885723:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830036.885734:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff8b2e88075a40 x1874672518194176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.885745:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518194176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.885754:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.885757:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830036.885760:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518194176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830036.885765:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518194304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830036.885781:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830036.885789:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830036/real 1787830036] req@ffff8b2e88077480 x1874672518194304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830036.885795:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518194304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830061 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830036.885804:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830036.885806:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830036.885808:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518194304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830037.204814:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830037.716774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830037.716865:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830037.908796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830037.908797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830038.228992:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830038.740810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830038.740870:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830038.933014:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830038.933595:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830039.252821:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830039.764858:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830039.764933:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830039.956786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830039.956901:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830040.276785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830040.788764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830040.788971:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830040.980774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830040.980899:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830041.300809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830041.812761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830041.812793:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830042.004766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830042.004766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830042.004796:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830042.004802:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830042.004807:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830042.004816:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3048 00000100:00080000:1.0:1787830042.004837:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830042.004844:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830042.004882:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518194432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830042.004912:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830042.004916:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830042.004918:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830042.004922:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3048 00000100:00080000:1.0:1787830042.004925:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830042.004928:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830042.004937:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830042.004950:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518194560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830042.004959:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830042.004961:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830042.004963:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830042.004966:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3048 00000100:00080000:1.0:1787830042.004969:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830042.004972:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830042.004976:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830042.004986:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518194688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830042.004994:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830042.004997:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830042.004999:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830042.005002:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3048 00000100:00080000:1.0:1787830042.005005:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830042.005008:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830042.005012:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830042.005021:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518194816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830042.005030:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830042.005032:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830042.005033:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830042.005037:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3048 00000100:00080000:1.0:1787830042.005039:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830042.005043:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830042.005047:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830042.005055:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518194944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830042.005061:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830042.005064:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830042.005066:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830042.005069:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3048 00000100:00080000:1.0:1787830042.005072:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830042.005074:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830042.005078:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830042.005087:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518195072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830042.005095:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830042.005098:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830042.005100:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830042.005103:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3048 00000100:00080000:1.0:1787830042.005106:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830042.005109:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830042.005113:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830042.005122:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518195200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830042.005197:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518194432:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830042.005250:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830042.005291:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830042/real 1787830042] req@ffff8b2e88076940 x1874672518194432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830042.005307:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518194432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830042.005316:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830042.005320:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830042.005324:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518194432:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830042.005331:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518194560:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830042.005352:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830042.005363:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830042/real 1787830042] req@ffff8b2e88077480 x1874672518194560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830042.005374:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518194560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830042.005382:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830042.005386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830042.005389:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518194560:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830042.005395:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518194688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830042.005413:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830042.005423:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830042/real 1787830042] req@ffff8b2e88075a40 x1874672518194688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830042.005434:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518194688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830042.005442:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830042.005445:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830042.005448:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518194688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830042.005454:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518194816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830042.005471:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830042.005480:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830042/real 1787830042] req@ffff8b2e88074f00 x1874672518194816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830042.005492:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518194816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830042.005499:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830042.005502:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830042.005505:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518194816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830042.005510:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518194944:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830042.005527:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830042.005537:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830042/real 1787830042] req@ffff8b2e88074780 x1874672518194944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830042.005550:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518194944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830042.005558:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830042.005561:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830042.005564:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518194944:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830042.005571:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518195072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830042.005587:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830042.005596:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830042/real 1787830042] req@ffff8b2e88075680 x1874672518195072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830042.005607:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518195072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830042.005615:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830042.005618:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830042.005621:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518195072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830042.005626:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518195200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830042.005644:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830042.005687:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830042/real 1787830042] req@ffff8b2e88077c00 x1874672518195200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830042.005698:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518195200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830067 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830042.005712:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830042.005715:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830042.005718:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518195200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830042.324745:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830042.836803:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830042.837061:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830043.028743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830043.028897:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830043.348781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830043.860794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830043.861182:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830044.052767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830044.052852:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830044.373356:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830044.884790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830044.884936:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830045.076931:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830045.077357:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830045.396803:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830045.909517:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830045.909642:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830046.100785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830046.100785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830046.420758:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830046.932916:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830046.932920:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830047.124805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787830047.124883:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830047.124888:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830047.124892:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830047.124900:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3053 00000100:00100000:3.0:1787830047.124904:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787830047.124909:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830047.124916:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830047.124949:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518195328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830047.124971:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830047.124973:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830047.124974:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830047.124977:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3053 00000100:00080000:2.0:1787830047.124980:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830047.124982:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830047.124990:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830047.124997:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518195456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830047.125002:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830047.125004:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830047.125005:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830047.125007:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3053 00000100:00080000:2.0:1787830047.125009:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830047.125010:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830047.125013:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830047.125017:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518195584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830047.125022:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830047.125023:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830047.125025:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830047.125027:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3053 00000100:00080000:2.0:1787830047.125028:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830047.125030:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830047.125032:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830047.125037:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518195712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830047.125042:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830047.125044:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830047.125045:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830047.125047:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3053 00000100:00080000:2.0:1787830047.125048:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830047.125050:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830047.125053:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830047.125060:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518195840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830047.125068:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830047.125070:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830047.125071:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830047.125074:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3053 00000100:00080000:2.0:1787830047.125077:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830047.125081:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830047.125084:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830047.125094:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518195968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830047.125103:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830047.125105:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830047.125107:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830047.125110:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3053 00000100:00080000:2.0:1787830047.125113:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830047.125116:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830047.125120:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830047.125129:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518196096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830047.125311:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518195328:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830047.125367:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830047.125409:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830047/real 1787830047] req@ffff8b2e8801e940 x1874672518195328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830047.125426:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518195328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830047.125435:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830047.125439:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830047.125443:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518195328:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830047.125449:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518195456:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830047.125470:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830047.125488:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830047/real 1787830047] req@ffff8b2e8801e580 x1874672518195456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830047.125501:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518195456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830047.125510:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830047.125513:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830047.125517:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518195456:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830047.125523:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518195584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830047.125541:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830047.125552:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830047/real 1787830047] req@ffff8b2e8801c000 x1874672518195584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830047.125563:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518195584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830047.125570:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830047.125573:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830047.125576:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518195584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830047.125581:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518195712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830047.125598:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830047.125609:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830047/real 1787830047] req@ffff8b2e8801cf00 x1874672518195712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830047.125621:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518195712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830047.125628:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830047.125630:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830047.125633:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518195712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830047.125638:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518195840:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830047.125677:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830047.125687:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830047/real 1787830047] req@ffff8b2e8801f480 x1874672518195840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830047.125699:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518195840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830047.125707:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830047.125710:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830047.125713:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518195840:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830047.125718:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518195968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830047.125735:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830047.125745:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830047/real 1787830047] req@ffff8b2e8801fc00 x1874672518195968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830047.125755:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518195968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830047.125763:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830047.125766:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830047.125769:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518195968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830047.125774:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518196096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830047.125803:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830047.125812:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830047/real 1787830047] req@ffff8b2e8801da40 x1874672518196096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830047.125823:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518196096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830072 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830047.125830:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830047.125833:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830047.125836:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518196096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830047.444852:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830047.956805:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830047.956885:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830048.149029:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830048.149126:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830048.468804:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830048.980969:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830048.981070:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830049.173104:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830049.173374:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830049.492802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830050.004805:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830050.004989:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830050.196720:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830050.196728:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830050.516777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830051.028786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830051.028859:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830051.220812:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830051.220931:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830051.540792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830052.052864:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830052.052926:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830052.244797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830052.244800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830052.244836:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830052.244856:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830052.244860:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830052.244871:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3058 00000100:00080000:3.0:1787830052.244883:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830052.244891:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830052.244938:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518196224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830052.245028:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518196224:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830052.245099:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830052.245130:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830052.245133:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830052.245134:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830052.245137:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3058 00000100:00080000:3.0:1787830052.245141:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830052.245145:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830052.245156:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830052.245170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518196352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830052.245691:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830052/real 1787830052] req@ffff8b2e8803b840 x1874672518196224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830052.245712:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518196224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830052.245723:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830052.245728:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830052.245732:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518196224:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830052.245769:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518196352:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830052.245796:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830052.245810:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830052/real 1787830052] req@ffff8b2e8803a580 x1874672518196352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830052.245822:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518196352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830052.245830:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830052.245833:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830052.245838:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518196352:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830052.245924:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830052.245928:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830052.245930:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830052.245935:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3058 00000100:00080000:3.0:1787830052.245939:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830052.245944:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830052.245951:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830052.245966:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518196480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830052.245977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830052.245980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830052.245982:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830052.245985:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3058 00000100:00080000:3.0:1787830052.245989:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830052.245993:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830052.245997:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830052.246005:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518196608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830052.246013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830052.246015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830052.246017:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830052.246021:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3058 00000100:00080000:3.0:1787830052.246024:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830052.246028:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830052.246033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830052.246044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518196736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830052.246052:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830052.246056:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830052.246058:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830052.246061:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3058 00000100:00080000:3.0:1787830052.246064:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830052.246068:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830052.246072:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830052.246081:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518196864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830052.246094:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830052.246097:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830052.246099:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830052.246102:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3058 00000100:00080000:3.0:1787830052.246106:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830052.246109:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830052.246113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830052.246122:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518196992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830052.246186:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518196480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830052.246211:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830052.246224:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830052/real 1787830052] req@ffff8b2e8803a1c0 x1874672518196480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830052.246234:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518196480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830052.246242:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830052.246245:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830052.246248:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518196480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830052.246254:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518196608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830052.246273:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830052.246284:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830052/real 1787830052] req@ffff8b2e88038f00 x1874672518196608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830052.246296:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518196608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830052.246303:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830052.246306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830052.246309:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518196608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830052.246315:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518196736:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830052.246334:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830052.246345:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830052/real 1787830052] req@ffff8b2e8803a580 x1874672518196736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830052.246352:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518196736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830052.246358:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830052.246360:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830052.246362:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518196736:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830052.246366:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518196864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830052.246378:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830052.246383:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830052/real 1787830052] req@ffff8b2e8803b840 x1874672518196864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830052.246390:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518196864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830052.246395:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830052.246396:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830052.246398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518196864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830052.246401:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518196992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830052.246410:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830052.246414:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830052/real 1787830052] req@ffff8b2e8803b0c0 x1874672518196992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830052.246421:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518196992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830077 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830052.246428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830052.246431:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830052.246434:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518196992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830052.564795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830053.076786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830053.076981:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830053.268780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830053.268780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830053.588719:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830054.100780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830054.100837:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830054.292762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830054.292762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830054.612782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830055.124768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830055.125046:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830055.316830:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830055.318709:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830055.636826:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830056.148744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830056.148898:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830056.340807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830056.340905:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830056.660806:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830057.172793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830057.172899:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830057.364803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830057.364803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830057.364895:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830057.364900:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830057.364904:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830057.364913:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3063 00000100:00080000:1.0:1787830057.364923:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830057.364929:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830057.364968:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518197120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830057.364995:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830057.364999:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830057.365000:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830057.365066:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518197120:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830057.365197:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3063 00000100:00080000:1.0:1787830057.365202:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830057.365206:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830057.365215:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830057.365237:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518197248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830057.365258:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830057.365261:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830057.365263:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830057.365267:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3063 00000100:00080000:1.0:1787830057.365270:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830057.365274:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830057.365278:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830057.365289:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518197376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830057.365298:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830057.365300:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830057.365302:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830057.365305:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3063 00000100:00080000:1.0:1787830057.365308:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830057.365310:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830057.365314:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830057.365323:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518197504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830057.365332:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830057.365335:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830057.365337:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830057.365424:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518197248:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830057.365455:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518197376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830057.365474:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518197504:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787830057.365521:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3063 00000100:00080000:1.0:1787830057.365525:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830057.365529:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830057.365533:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830057.365544:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518197632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830057.365586:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518197632:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830057.365624:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830057.365627:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830057.365629:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830057.365633:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3063 00000100:00080000:1.0:1787830057.365636:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830057.365639:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830057.365643:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830057.365699:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518197760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830057.369987:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518197760:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787830057.370237:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830057.370243:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830057.370247:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830057.370253:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3063 00000100:00080000:1.0:1787830057.370259:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830057.370266:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830057.370275:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830057.370303:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518197888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830057.370453:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518197888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830057.685376:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830058.197101:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830058.201012:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830058.388795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830058.389231:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830058.708776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830059.224773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830059.224908:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830059.412810:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830059.412931:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830059.733375:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830060.244915:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830060.244979:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830060.437001:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830060.437030:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:1.0:1787830060.437159:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787830060.437202:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787830060.437219:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830060.437265:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830060.437273:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830060.437281:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830060.437288:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830060.437295:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830060.437301:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787830060.437341:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830057/real 1787830060] req@ffff8b2e88076d00 x1874672518197888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830060.437363:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518197888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830060.437380:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830060.437384:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830060.437389:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518197888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830060.437395:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830057/real 1787830060] req@ffff8b2e88077840 x1874672518197760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830060.437408:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518197760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830060.437415:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830060.437418:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830060.437421:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518197760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830060.437427:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830057/real 1787830060] req@ffff8b2e880770c0 x1874672518197632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830060.437448:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518197632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830060.437470:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830060.437474:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830060.437536:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518197632:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830060.437543:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830057/real 1787830060] req@ffff8b2e880761c0 x1874672518197248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830060.437556:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518197248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830060.437563:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830060.437683:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830060.437697:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518197248:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830060.437705:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830057/real 1787830060] req@ffff8b2e88074000 x1874672518197376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830060.437718:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518197376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830060.437726:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830060.437729:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830060.437732:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518197376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830060.437738:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830057/real 1787830060] req@ffff8b2e88074b40 x1874672518197504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830060.437749:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518197504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830060.437755:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830060.437758:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830060.437761:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518197504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830060.437767:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830057/real 1787830060] req@ffff8b2e88075e00 x1874672518197120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830060.437777:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518197120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830082 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830060.437784:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830060.437787:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830060.437790:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518197120:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830060.756748:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830061.268789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830061.268822:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830061.461300:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830061.461386:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830061.781297:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830062.293353:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830062.293378:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:1.0:1787830062.484777:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830062.484787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830062.484789:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830062.484795:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830062.484801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830062.484806:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3068 00000100:00080000:1.0:1787830062.484817:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830062.484824:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830062.484916:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518198016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830062.484940:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830062.484944:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830062.484945:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830062.484949:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3068 00000100:00080000:1.0:1787830062.484954:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830062.484958:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830062.484969:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830062.484973:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518198016:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830062.484985:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518198144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830062.484996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830062.485000:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830062.485002:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830062.485005:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3068 00000100:00080000:1.0:1787830062.485022:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830062.485026:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787830062.485032:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787830062.485032:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830062.485048:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518198272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830062.485058:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830062.485061:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830062.485063:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830062.485067:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3068 00000100:00080000:1.0:1787830062.485071:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830062.485073:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830062/real 1787830062] req@ffff8b2e88076580 x1874672518198016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830062.485077:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830062.485085:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830062.485097:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518198016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830062.485101:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518198400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830062.485119:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830062.485122:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830062.485126:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787830062.485127:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830062.485132:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830062.485134:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518198016:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830062.485158:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518198144:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830062.485181:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830062.485194:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830062/real 1787830062] req@ffff8b2e88075e00 x1874672518198144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830062.485206:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518198144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830062.485214:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830062.485218:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830062.485221:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518198144:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830062.485227:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518198272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830062.485248:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830062.485259:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830062/real 1787830062] req@ffff8b2e88074b40 x1874672518198272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830062.485271:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518198272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830062.485278:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830062.485281:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830062.485284:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518198272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830062.485289:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518198400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830062.485307:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830062.485317:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830062/real 1787830062] req@ffff8b2e88074000 x1874672518198400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830062.485328:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518198400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830062.485336:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830062.485339:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830062.485342:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518198400:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787830062.486776:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3068 00000100:00080000:1.0:1787830062.486787:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830062.486794:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830062.486805:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830062.486833:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518198528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830062.486953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830062.486958:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830062.486960:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830062.486965:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3068 00000100:00080000:1.0:1787830062.486969:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830062.486973:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830062.486978:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830062.486991:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518198656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830062.487000:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830062.487002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830062.487004:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830062.487007:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3068 00000100:00080000:1.0:1787830062.487010:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830062.487013:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830062.487018:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830062.487030:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518198784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830062.487181:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518198528:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830062.487227:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830062.487249:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830062/real 1787830062] req@ffff8b2e880761c0 x1874672518198528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830062.487271:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518198528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830062.487280:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830062.487284:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830062.487287:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518198528:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830062.487294:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518198656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830062.487315:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830062.487326:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830062/real 1787830062] req@ffff8b2e880770c0 x1874672518198656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830062.487337:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518198656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830062.487344:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830062.487347:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830062.487350:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518198656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830062.487356:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518198784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830062.487374:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830062.487384:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830062/real 1787830062] req@ffff8b2e88077840 x1874672518198784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830062.487395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518198784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830087 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830062.487408:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830062.487411:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830062.487413:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518198784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830062.804792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830063.316786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830063.316942:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830063.508785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830063.508896:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830063.828771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830064.340794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830064.340820:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830064.532761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830064.533134:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830064.852761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830065.365095:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830065.365848:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830065.556830:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830065.557385:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830065.877211:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830066.390895:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830066.391056:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830066.580792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830066.581034:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830066.900797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830067.412847:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830067.412969:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830067.604770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830067.607170:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830067.607182:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830067.607186:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830067.607197:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3074 00000100:00080000:1.0:1787830067.607206:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830067.607213:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830067.607251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518198912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830067.607292:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830067.607296:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830067.607298:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830067.607302:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3074 00000100:00080000:1.0:1787830067.607305:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830067.607309:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830067.607318:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830067.607331:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518199040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830067.607341:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830067.607344:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830067.607345:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830067.607349:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3074 00000100:00080000:1.0:1787830067.607352:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830067.607355:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830067.607359:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830067.607368:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518199168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830067.607376:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830067.607379:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830067.607380:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830067.607384:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3074 00000100:00080000:1.0:1787830067.607386:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830067.607389:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830067.607393:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830067.607401:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518199296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830067.607409:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830067.607411:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830067.607413:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830067.607417:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3074 00000100:00080000:1.0:1787830067.607420:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830067.607423:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830067.607427:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830067.607435:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518199424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830067.607443:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830067.607446:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830067.607447:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830067.607451:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3074 00000100:00080000:1.0:1787830067.607453:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830067.607456:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830067.607460:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830067.607468:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518199552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830067.607477:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830067.607480:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830067.607481:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830067.607485:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3074 00000100:00080000:1.0:1787830067.607488:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830067.607491:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830067.607495:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830067.607505:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518199680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830067.607760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830067.608301:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518198912:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830067.608368:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830067.608420:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830067/real 1787830067] req@ffff8b2e88076d00 x1874672518198912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830067.608438:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518198912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830067.608448:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830067.608454:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830067.608458:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518198912:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830067.608465:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518199040:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830067.608488:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830067.608502:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830067/real 1787830067] req@ffff8b2e88077c00 x1874672518199040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830067.608512:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518199040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830067.608520:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830067.608523:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830067.608526:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518199040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830067.608533:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518199168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830067.608552:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830067.608563:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830067/real 1787830067] req@ffff8b2e88075680 x1874672518199168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830067.608575:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518199168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830067.608582:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830067.608586:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830067.608589:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518199168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830067.608594:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518199296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830067.608613:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830067.608622:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830067/real 1787830067] req@ffff8b2e88074780 x1874672518199296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830067.608633:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518199296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830067.608683:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830067.608687:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830067.608690:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518199296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830067.608696:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518199424:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830067.608716:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830067.608727:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830067/real 1787830067] req@ffff8b2e88074f00 x1874672518199424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830067.608738:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518199424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830067.608746:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830067.608749:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830067.608753:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518199424:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830067.608758:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518199552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830067.608775:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830067.608784:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830067/real 1787830067] req@ffff8b2e88075a40 x1874672518199552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830067.608801:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518199552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830067.608808:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830067.608812:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830067.608814:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518199552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830067.608820:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518199680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830067.608835:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830067.608843:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830067/real 1787830067] req@ffff8b2e88077480 x1874672518199680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830067.608852:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518199680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830092 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830067.608859:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830067.608862:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830067.608865:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518199680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830067.924778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830068.437012:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830068.437066:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830068.629079:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830068.630544:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830068.948776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830069.460889:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830069.463829:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830069.652787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830069.652898:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830069.972782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830070.484812:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830070.485020:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830070.677206:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830070.677206:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830070.997038:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830071.508781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830071.508941:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830071.700865:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830071.701021:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830072.022958:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830072.533684:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830072.533983:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830072.724803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830072.724908:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830072.725796:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830072.725802:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830072.725806:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830072.725815:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3079 00000100:00080000:3.0:1787830072.725825:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830072.725832:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830072.725867:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518199808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830072.725901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830072.725906:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830072.725908:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830072.725912:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3079 00000100:00080000:3.0:1787830072.725916:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830072.725919:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830072.725929:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830072.725941:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518199936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830072.725951:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830072.725954:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830072.725956:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830072.725959:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3079 00000100:00080000:3.0:1787830072.725963:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830072.725966:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830072.725970:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830072.725980:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518200064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830072.725989:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830072.725992:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830072.725994:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830072.725997:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3079 00000100:00080000:3.0:1787830072.726000:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830072.726003:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830072.726007:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830072.726017:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518200192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830072.726026:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830072.726028:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830072.726030:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830072.726033:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3079 00000100:00100000:1.0:1787830072.726036:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518199808:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830072.726036:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830072.726039:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830072.726043:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830072.726051:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518200320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830072.726060:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830072.726062:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830072.726064:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830072.726067:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3079 00000100:00080000:3.0:1787830072.726069:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830072.726072:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830072.726075:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830072.726083:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518200448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830072.726091:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830072.726093:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830072.726094:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830072.726097:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3079 00000100:00080000:3.0:1787830072.726100:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830072.726102:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:1.0:1787830072.726103:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:3.0:1787830072.726106:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830072.726116:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518200576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830072.726156:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830072/real 1787830072] req@ffff8b2e880383c0 x1874672518199808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830072.726170:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518199808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830072.726180:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830072.726190:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830072.726194:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518199808:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830072.726200:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518199936:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830072.726231:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830072.726245:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830072/real 1787830072] req@ffff8b2e88039680 x1874672518199936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830072.726264:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518199936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830072.726272:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830072.726275:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830072.726279:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518199936:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830072.726284:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518200064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830072.727141:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830072.727166:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830072/real 1787830072] req@ffff8b2e880392c0 x1874672518200064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830072.727186:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518200064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830072.727195:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830072.727198:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830072.727201:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518200064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830072.727208:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518200192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830072.727238:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830072.727256:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830072/real 1787830072] req@ffff8b2e8803b480 x1874672518200192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830072.727268:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518200192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830072.727275:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830072.727278:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830072.727288:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518200192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830072.727321:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518200320:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830072.727408:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830072.727421:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830072/real 1787830072] req@ffff8b2e88039e00 x1874672518200320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830072.727432:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518200320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830072.727441:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830072.727444:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830072.727447:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518200320:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830072.727453:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518200448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830072.727483:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830072.727495:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830072/real 1787830072] req@ffff8b2e88039a40 x1874672518200448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830072.727551:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518200448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830072.727558:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830072.727561:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830072.727564:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518200448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830072.727578:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518200576:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830072.727598:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830072.727625:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830072/real 1787830072] req@ffff8b2e8803b0c0 x1874672518200576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830072.727637:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518200576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830097 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830072.727644:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830072.727668:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830072.727673:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518200576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830073.045676:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830073.557453:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830073.558042:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830073.749086:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830073.749498:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830074.068741:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830074.580774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830074.580984:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830074.773060:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830074.773204:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830075.093069:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830075.605629:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830075.605820:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830075.797279:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830075.797358:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830076.121211:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830076.628763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830076.628794:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830076.820933:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830076.821072:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830077.143832:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830077.652929:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830077.653140:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830077.845665:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830077.845738:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830077.845744:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830077.845748:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830077.845756:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3084 00000100:00080000:0.0:1787830077.845767:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830077.845774:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830077.845813:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518200704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830077.845871:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518200704:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830077.845926:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830077.845961:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830077.845965:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830077.845967:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830077.845972:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3084 00000100:00080000:0.0:1787830077.845976:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830077.845981:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830077.845991:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830077.846006:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518200832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830077.846069:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830077/real 1787830077] req@ffff8b2e88041e00 x1874672518200704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830077.846084:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518200704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830077.846093:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830077.846097:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830077.846101:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518200704:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830077.846123:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518200832:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830077.846146:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830077.846159:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830077/real 1787830077] req@ffff8b2e88042580 x1874672518200832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830077.846171:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518200832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830077.846186:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830077.846189:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830077.846192:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518200832:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787830077.846285:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830077.846289:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830077.846291:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830077.846295:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3084 00000100:00080000:0.0:1787830077.846300:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830077.846303:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830077.846309:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830077.846322:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518200960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830077.846367:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518200960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830077.846391:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830077.847236:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830077.847621:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830077.847625:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830077.847627:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830077.847632:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3084 00000100:00080000:0.0:1787830077.847637:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830077.847640:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830077.847657:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830077.847673:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518201088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830077.847739:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830077/real 1787830077] req@ffff8b2e88042d00 x1874672518200960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830077.847753:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518200960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830077.847761:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830077.847764:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830077.847768:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518200960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830077.847782:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518201088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830077.847805:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830077.847817:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830077/real 1787830077] req@ffff8b2e88040780 x1874672518201088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830077.847828:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518201088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830077.847836:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830077.847839:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830077.847842:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518201088:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787830077.847967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830077.847972:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830077.847974:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830077.847979:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3084 00000100:00080000:0.0:1787830077.847983:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830077.847986:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830077.847992:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830077.848004:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518201216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830077.848047:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518201216:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830077.848068:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830077.848088:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830077.848092:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830077.848094:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830077.848098:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3084 00000100:00080000:0.0:1787830077.848102:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830077.848105:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830077.848110:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830077.848121:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518201344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830077.848159:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830077/real 1787830077] req@ffff8b2e88043840 x1874672518201216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830077.848172:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518201216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830077.848180:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830077.848183:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830077.848186:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518201216:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830077.848198:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518201344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830077.848217:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830077.848229:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830077/real 1787830077] req@ffff8b2e880412c0 x1874672518201344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830077.848240:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518201344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830077.848248:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830077.848251:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830077.848254:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518201344:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787830077.848331:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830077.848335:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830077.848337:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830077.848341:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3084 00000100:00080000:0.0:1787830077.848345:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830077.848348:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830077.848353:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830077.848366:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518201472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830077.848409:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518201472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830077.848428:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830077.848485:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830077/real 1787830077] req@ffff8b2e88040b40 x1874672518201472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830077.848497:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518201472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830102 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830077.848505:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830077.848508:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830077.848511:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518201472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830078.165020:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830078.677045:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830078.677629:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830078.868780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830078.868780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830079.188796:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830079.701463:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830079.701779:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830079.892842:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830079.893236:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830080.213061:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830080.725625:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830080.725693:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830080.916787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830080.918137:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830081.236864:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830081.748780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830081.761445:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830081.940748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830081.940748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830082.261017:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830082.773171:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830082.773784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830082.964790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830082.967136:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830082.970074:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830082.970087:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830082.970092:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830082.970112:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3089 00000100:00080000:1.0:1787830082.970124:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830082.970131:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830082.970170:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518201600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830082.970197:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830082.970200:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830082.970202:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830082.970206:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3089 00000100:00080000:1.0:1787830082.970210:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830082.970213:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830082.970223:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830082.970238:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518201728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830082.970247:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830082.970250:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830082.970251:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830082.970255:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3089 00000100:00080000:1.0:1787830082.970258:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830082.970261:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830082.970266:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830082.970275:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518201856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830082.970283:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830082.970286:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830082.970288:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830082.970291:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3089 00000100:00080000:1.0:1787830082.970294:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830082.970297:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830082.970301:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830082.970311:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518201984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830082.970319:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830082.970322:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830082.970323:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830082.970326:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3089 00000100:00080000:1.0:1787830082.970329:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830082.970332:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830082.970336:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830082.970345:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518202112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830082.970353:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830082.970355:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830082.970357:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830082.970360:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3089 00000100:00080000:1.0:1787830082.970363:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830082.970366:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830082.970369:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830082.970481:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518202240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830082.970491:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830082.970494:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830082.970496:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830082.970500:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3089 00000100:00080000:1.0:1787830082.970504:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830082.970507:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830082.970512:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830082.970521:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518202368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830082.971827:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518201600:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830082.971908:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830082.971955:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830082/real 1787830082] req@ffff8b2e88076940 x1874672518201600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830082.971972:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518201600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830082.971982:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830082.971986:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830082.971989:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518201600:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830082.971996:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518201728:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830082.972019:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830082.972033:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830082/real 1787830082] req@ffff8b2e88077480 x1874672518201728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830082.972044:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518201728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830082.972052:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830082.972055:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830082.972058:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518201728:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830082.972065:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518201856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830082.972086:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830082.972097:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830082/real 1787830082] req@ffff8b2e88075a40 x1874672518201856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830082.972108:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518201856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830082.972116:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830082.972119:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830082.972122:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518201856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830082.972129:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518201984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830082.972147:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830082.972158:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830082/real 1787830082] req@ffff8b2e88074f00 x1874672518201984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830082.972168:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518201984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830082.972175:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830082.972178:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830082.972180:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518201984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830082.972185:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518202112:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830082.972203:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830082.972213:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830082/real 1787830082] req@ffff8b2e88074780 x1874672518202112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830082.972244:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518202112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830082.972253:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830082.972256:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830082.972260:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518202112:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830082.972266:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518202240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830082.972286:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830082.972297:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830082/real 1787830082] req@ffff8b2e88075680 x1874672518202240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830082.972308:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518202240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830082.972315:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830082.972319:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830082.972321:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518202240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830082.972327:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518202368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830082.972345:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830082.972356:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830082/real 1787830082] req@ffff8b2e88077c00 x1874672518202368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830082.972367:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518202368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830107 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830082.972380:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830082.972383:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830082.972386:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518202368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830083.284850:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830083.796718:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830083.796952:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830083.988793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830083.989002:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830084.308724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830084.820954:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830084.839141:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830085.012776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830085.012776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830085.334135:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830085.844773:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830085.844854:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830086.036781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830086.036841:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830086.356823:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830086.868779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830086.869582:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830087.060721:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830087.061358:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830087.381560:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830087.893781:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830087.894083:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830088.084787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830088.085034:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830088.085039:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830088.085043:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830088.085050:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3094 00000100:00100000:1.0:1787830088.085059:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830088.085060:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830088.085068:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830088.085100:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518202496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830088.085123:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830088.085125:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830088.085126:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830088.085130:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3094 00000100:00080000:0.0:1787830088.085132:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830088.085134:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830088.085142:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830088.085149:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518202624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830088.085155:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830088.085157:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830088.085158:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830088.085160:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3094 00000100:00080000:0.0:1787830088.085161:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830088.085163:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830088.085166:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830088.085174:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518202752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830088.085179:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830088.085181:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830088.085182:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830088.085184:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3094 00000100:00080000:0.0:1787830088.085186:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830088.085188:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830088.085191:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830088.085196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518202880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830088.085202:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830088.085204:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830088.085206:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830088.085209:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3094 00000100:00080000:0.0:1787830088.085212:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830088.085214:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830088.085217:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830088.085222:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518203008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830088.085239:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830088.085242:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830088.085245:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830088.085248:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3094 00000100:00080000:0.0:1787830088.085251:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830088.085255:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830088.085259:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830088.085269:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518203136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830088.085277:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830088.085279:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830088.085281:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830088.085284:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3094 00000100:00080000:0.0:1787830088.085287:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830088.085290:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830088.085294:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830088.085305:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518203264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830088.085629:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518202496:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830088.085706:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830088.085903:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830088/real 1787830088] req@ffff8b2e880421c0 x1874672518202496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830088.085924:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518202496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830088.085934:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830088.085938:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830088.085942:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518202496:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830088.085950:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518202624:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830088.086046:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830088.086062:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830088/real 1787830088] req@ffff8b2e88041a40 x1874672518202624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830088.086075:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518202624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830088.086083:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830088.086086:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830088.086090:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518202624:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830088.086096:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518202752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830088.086115:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830088.086186:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830088/real 1787830088] req@ffff8b2e88040f00 x1874672518202752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830088.086202:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518202752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830088.086209:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830088.086212:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830088.086215:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518202752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830088.086222:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518202880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830088.086244:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830088.086256:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830088/real 1787830088] req@ffff8b2e88042940 x1874672518202880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830088.086268:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518202880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830088.086275:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830088.086278:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830088.086281:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518202880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830088.086287:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518203008:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830088.086308:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830088.086318:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830088/real 1787830088] req@ffff8b2e88043480 x1874672518203008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830088.086329:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518203008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830088.086336:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830088.086339:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830088.086342:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518203008:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830088.086347:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518203136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830088.086362:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830088.086370:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830088/real 1787830088] req@ffff8b2e880430c0 x1874672518203136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830088.086380:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518203136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830088.086388:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830088.086390:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830088.086393:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518203136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830088.086398:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518203264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830088.086413:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830088.086422:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830088/real 1787830088] req@ffff8b2e88040000 x1874672518203264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830088.086432:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518203264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830113 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830088.086440:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830088.086442:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830088.086445:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518203264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830088.404907:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830088.917048:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830088.917502:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830089.108781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830089.108781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830089.428791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830089.940774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830089.940817:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830090.132775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830090.132775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830090.454096:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830090.964780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830090.964914:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830091.156811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830091.157213:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830091.476801:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830091.988791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830091.991971:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830092.180834:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830092.180976:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830092.500772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830093.012906:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830093.013349:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:0.0:1787830093.204884:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830093.204896:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830093.204899:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830093.204909:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3099 00000100:00080000:0.0:1787830093.204920:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830093.204927:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830093.204965:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518203392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830093.204988:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830093.204994:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830093.204998:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830093.205000:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830093.205005:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3099 00000100:00080000:0.0:1787830093.205008:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830093.205012:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830093.205023:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830093.205037:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518203520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830093.205046:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830093.205049:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830093.205051:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830093.205054:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3099 00000100:00080000:0.0:1787830093.205057:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830093.205059:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830093.205062:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830093.205070:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518203648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830093.205078:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830093.205080:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830093.205082:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830093.205085:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3099 00000100:00080000:0.0:1787830093.205088:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830093.205090:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830093.205093:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830093.205101:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518203776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830093.205107:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830093.205110:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830093.205112:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830093.205115:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3099 00000100:00080000:0.0:1787830093.205118:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830093.205121:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830093.205125:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830093.205132:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518203904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830093.205138:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830093.205141:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830093.205143:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830093.205146:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3099 00000100:00080000:0.0:1787830093.205148:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830093.205150:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830093.205153:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830093.205161:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518204032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830093.205168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830093.205170:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830093.205172:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830093.205175:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3099 00000100:00080000:0.0:1787830093.205177:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830093.205180:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830093.205184:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830093.205192:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518204160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830093.205243:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518203392:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830093.205299:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830093.205339:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830093/real 1787830093] req@ffff8b2e880403c0 x1874672518203392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830093.205354:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518203392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830093.205363:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830093.205366:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830093.205370:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518203392:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830093.205376:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518203520:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830093.205392:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830093.205402:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830093/real 1787830093] req@ffff8b2e88040000 x1874672518203520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830093.205412:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518203520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830093.205420:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830093.205423:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830093.205426:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518203520:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830093.205431:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518203648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830093.205443:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830093.205451:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830093/real 1787830093] req@ffff8b2e880430c0 x1874672518203648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830093.205516:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830093.205539:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518203648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830093.205548:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830093.205551:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830093.205554:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518203648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830093.205559:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518203776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830093.205576:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830093.205585:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830093/real 1787830093] req@ffff8b2e88043480 x1874672518203776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830093.205595:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518203776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830093.205603:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830093.205605:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830093.205608:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518203776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830093.205613:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518203904:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830093.205625:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830093.205633:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830093/real 1787830093] req@ffff8b2e88042940 x1874672518203904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830093.205642:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518203904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830093.205689:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830093.205692:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830093.205695:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518203904:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830093.205701:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518204032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830093.205716:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830093.207035:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830093/real 1787830093] req@ffff8b2e88040f00 x1874672518204032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830093.207052:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518204032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830093.207060:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830093.207064:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830093.207067:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518204032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830093.207073:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518204160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830093.207100:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830093.207111:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830093/real 1787830093] req@ffff8b2e88041a40 x1874672518204160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830093.207122:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518204160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830118 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830093.207129:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830093.207132:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830093.207135:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518204160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830093.526091:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830094.036947:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830094.037522:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830094.228840:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830094.228875:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830094.549389:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830095.060842:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830095.061162:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830095.252771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830095.252825:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830095.572793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830096.084754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830096.085148:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830096.276797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830096.276817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830096.596817:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830097.108780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830097.108883:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830097.301088:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830097.301298:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830097.621022:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830098.132763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830098.132795:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830098.324804:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830098.324873:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830098.324880:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830098.324884:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830098.324893:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3104 00000100:00080000:0.0:1787830098.324904:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830098.324911:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830098.324947:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518204288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830098.325110:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518204288:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830098.325169:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830098.325183:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830098.325318:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830098.325322:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830098.325324:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830098.325328:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3104 00000100:00080000:0.0:1787830098.325333:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830098.325338:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830098.325347:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830098.325361:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518204416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830098.325593:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830098.325596:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830098.325598:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830098.325601:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3104 00000100:00080000:0.0:1787830098.325604:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830098.325607:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830098.325611:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830098.325623:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518204544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830098.325628:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830098/real 1787830098] req@ffff8b2e880421c0 x1874672518204288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830098.325631:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830098.325635:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830098.325637:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830098.325645:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3104 00000100:00080000:0.0:1787830098.325665:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830098.325667:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830098.325670:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830098.325678:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518204672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830098.325689:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830098.325694:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00100000:3.0:1787830098.325695:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518204288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830098.325697:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830098.325706:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3104 00000100:00080000:3.0:1787830098.325714:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830098.325720:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830098.325723:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518204288:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830098.325745:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518204416:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787830098.325818:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830098.325824:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830098.325829:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830098.325841:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518204800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830098.325851:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830098.325854:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830098.325856:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830098.325859:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3104 00000100:00080000:0.0:1787830098.325894:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830098.325901:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787830098.325901:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787830098.325909:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830098.325924:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518204928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830098.325924:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830098/real 1787830098] req@ffff8b2e88040b40 x1874672518204416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830098.325955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830098.325959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787830098.325961:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518204416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830098.325962:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830098.325968:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3104 00000100:00080000:0.0:1787830098.325973:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830098.325979:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787830098.325981:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787830098.325985:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787830098.325989:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830098.325994:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518204416:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830098.326000:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518205056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830098.326004:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518204544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830098.326036:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830098.326048:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830098/real 1787830098] req@ffff8b2e880412c0 x1874672518204544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830098.326059:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518204544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830098.326066:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830098.326069:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830098.326072:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518204544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830098.326077:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518204672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830098.326095:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830098.326106:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830098/real 1787830098] req@ffff8b2e88043840 x1874672518204672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830098.326117:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518204672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830098.326124:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830098.326127:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830098.326130:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518204672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830098.326144:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518204800:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830098.326161:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830098.326171:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830098/real 1787830098] req@ffff8b2e88040780 x1874672518204800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830098.326182:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518204800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830098.326190:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830098.326192:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830098.326195:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518204800:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830098.326200:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518204928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830098.326215:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830098.326223:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830098/real 1787830098] req@ffff8b2e88042d00 x1874672518204928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830098.326233:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518204928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830098.326240:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830098.326243:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830098.326246:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518204928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830098.326251:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518205056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830098.326267:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830098.326276:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830098/real 1787830098] req@ffff8b2e88042580 x1874672518205056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830098.326292:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518205056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830123 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830098.326299:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830098.326301:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830098.326304:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518205056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830098.644987:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830099.156735:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830099.156821:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830099.348783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830099.348854:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830099.671225:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830100.180760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830100.180868:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830100.372815:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830100.372876:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830100.692777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830101.204778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830101.204834:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830101.397052:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830101.397087:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830101.717320:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830102.228829:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830102.228860:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830102.421449:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830102.421554:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830102.740879:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830103.252845:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830103.253071:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830103.444780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830103.444780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830103.444915:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830103.444922:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830103.444925:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830103.444946:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3109 00000100:00080000:0.0:1787830103.444956:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830103.444964:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830103.445127:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518205184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830103.445154:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830103.445157:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830103.445158:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830103.445161:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3109 00000100:00080000:0.0:1787830103.445164:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830103.445167:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830103.445176:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830103.445194:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518205312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830103.445200:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830103.445203:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830103.445204:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830103.445206:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3109 00000100:00080000:0.0:1787830103.445208:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830103.445210:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830103.445213:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830103.445219:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518205440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830103.445225:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830103.445227:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830103.445228:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830103.445229:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3109 00000100:00080000:0.0:1787830103.445231:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830103.445232:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830103.445234:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830103.445239:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518205568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830103.445244:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830103.445245:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830103.445246:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830103.445249:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3109 00000100:00080000:0.0:1787830103.445250:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830103.445252:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830103.445255:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830103.445259:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518205696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830103.445263:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830103.445265:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830103.445266:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830103.445268:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3109 00000100:00080000:0.0:1787830103.445270:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830103.445272:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830103.445274:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830103.445279:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518205824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830103.445283:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830103.445284:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830103.445285:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830103.445287:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3109 00000100:00080000:0.0:1787830103.445288:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830103.445290:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830103.445292:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830103.445297:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518205952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830103.445614:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518205184:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830103.445693:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830103.448826:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830103/real 1787830103] req@ffff8b2e88041e00 x1874672518205184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830103.448858:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518205184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830103.448869:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830103.448873:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830103.448877:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518205184:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830103.448888:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518205312:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830103.448945:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830103.448967:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830103/real 1787830103] req@ffff8b2e88042580 x1874672518205312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830103.448979:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518205312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830103.448987:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830103.448991:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830103.448994:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518205312:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830103.449000:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518205440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830103.449021:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830103.449032:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830103/real 1787830103] req@ffff8b2e88042d00 x1874672518205440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830103.449043:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518205440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830103.449050:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830103.449053:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830103.449056:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518205440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830103.449061:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518205568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830103.449079:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830103.449089:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830103/real 1787830103] req@ffff8b2e88040780 x1874672518205568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830103.449107:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518205568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830103.449114:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830103.449117:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830103.449120:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518205568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830103.449126:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518205696:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830103.449145:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830103.449155:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830103/real 1787830103] req@ffff8b2e88043840 x1874672518205696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830103.449165:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518205696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830103.449172:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830103.449175:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830103.449178:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518205696:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830103.449184:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518205824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830103.449202:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830103.449212:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830103/real 1787830103] req@ffff8b2e880412c0 x1874672518205824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830103.449222:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518205824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830103.449233:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830103.449236:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830103.449239:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518205824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830103.449245:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518205952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830103.449263:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830103.449273:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830103/real 1787830103] req@ffff8b2e88040b40 x1874672518205952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830103.449283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518205952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830128 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830103.449290:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830103.449293:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830103.449295:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518205952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830103.765165:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830104.276836:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830104.276877:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830104.468808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830104.469000:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830104.788742:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830105.300866:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830105.301498:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830105.492785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830105.492845:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830105.812909:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830106.324862:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830106.324881:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830106.516777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830106.517437:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830106.836731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830107.348884:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830107.348950:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830107.541487:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830107.543825:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830107.860782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830108.372765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830108.372835:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830108.564764:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830108.564775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830108.564777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830108.564777:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830108.564782:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830108.564793:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3114 00000100:00080000:1.0:1787830108.564804:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830108.564811:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830108.564849:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518206080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830108.564880:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830108.564883:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830108.564885:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830108.564889:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3114 00000100:00080000:1.0:1787830108.564894:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830108.564897:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830108.564907:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830108.564920:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518206208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830108.564930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830108.564943:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830108.564945:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830108.564949:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3114 00000100:00080000:1.0:1787830108.564953:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830108.564957:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830108.564962:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830108.564972:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518206336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830108.564981:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830108.564984:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830108.564986:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830108.564989:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3114 00000100:00080000:1.0:1787830108.564992:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830108.564996:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830108.565000:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830108.565011:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518206464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830108.565019:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830108.565022:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830108.565024:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830108.565028:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3114 00000100:00080000:1.0:1787830108.565030:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830108.565034:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830108.565038:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830108.565048:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518206592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830108.565057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830108.565060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830108.565062:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830108.565065:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3114 00000100:00080000:1.0:1787830108.565068:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830108.565071:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830108.565075:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830108.565085:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518206720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830108.565107:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830108.565110:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830108.565112:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830108.565116:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3114 00000100:00080000:1.0:1787830108.565119:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830108.565123:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830108.565127:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830108.565136:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518206848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830108.565220:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518206080:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830108.565273:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830108.565316:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830108/real 1787830108] req@ffff8b2e88076d00 x1874672518206080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830108.565338:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518206080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830108.565347:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830108.565350:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830108.565366:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518206080:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830108.565374:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518206208:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830108.565395:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830108.565407:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830108/real 1787830108] req@ffff8b2e88077840 x1874672518206208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830108.565418:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518206208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830108.565426:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830108.565429:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830108.565432:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518206208:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830108.565437:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518206336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830108.565456:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830108.565467:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830108/real 1787830108] req@ffff8b2e880770c0 x1874672518206336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830108.565484:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518206336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830108.565496:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830108.565500:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830108.565503:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518206336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830108.565508:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518206464:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830108.565526:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830108.565537:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830108/real 1787830108] req@ffff8b2e880761c0 x1874672518206464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830108.565548:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518206464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830108.565555:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830108.565558:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830108.565561:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518206464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830108.565566:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518206592:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830108.565582:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830108.565591:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830108/real 1787830108] req@ffff8b2e88074000 x1874672518206592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830108.565602:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518206592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830108.565609:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830108.565612:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830108.565615:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518206592:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830108.565621:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518206720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830108.565675:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830108.565688:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830108/real 1787830108] req@ffff8b2e88074b40 x1874672518206720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830108.565700:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518206720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830108.565707:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830108.565710:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830108.565713:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518206720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830108.565719:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518206848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830108.565738:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830108.565747:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830108/real 1787830108] req@ffff8b2e88075e00 x1874672518206848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830108.565758:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518206848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830133 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830108.565766:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830108.565769:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830108.565772:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518206848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830108.884879:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830109.397074:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830109.397455:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830109.589000:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830109.589227:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830109.908886:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830110.420834:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830110.421527:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830110.612771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830110.614788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830110.933098:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830111.444800:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830111.444872:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830111.636902:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830111.637010:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830111.956829:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830112.468853:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830112.469590:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830112.660782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830112.661027:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830112.980768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830113.492749:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830113.493205:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830113.684917:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830113.685044:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830113.685451:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830113.685458:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830113.685462:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830113.685471:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3120 00000100:00080000:3.0:1787830113.685481:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830113.685488:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830113.685525:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518206976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830113.685556:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830113.685559:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830113.685561:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830113.685565:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3120 00000100:00080000:3.0:1787830113.685568:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830113.685571:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830113.685580:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830113.685590:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518207104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830113.685598:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830113.685601:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830113.685603:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830113.685606:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3120 00000100:00080000:3.0:1787830113.685609:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830113.685612:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830113.685615:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830113.685622:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518207232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830113.685629:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830113.685632:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830113.685633:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830113.685636:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3120 00000100:00080000:3.0:1787830113.685638:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830113.685640:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830113.685644:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830113.685676:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518207360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830113.685684:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830113.685687:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830113.685689:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830113.685692:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3120 00000100:00080000:3.0:1787830113.685695:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830113.685698:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830113.685702:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830113.685713:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518207488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830113.685721:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830113.685724:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830113.685725:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830113.685728:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3120 00000100:00080000:3.0:1787830113.685731:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830113.685733:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830113.685737:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830113.685745:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518207616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830113.685753:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830113.685755:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830113.685757:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830113.685760:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3120 00000100:00080000:3.0:1787830113.685763:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830113.685765:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830113.685769:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830113.685777:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518207744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830113.686022:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518206976:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830113.686082:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830113.686124:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830113/real 1787830113] req@ffff8b2e8803b840 x1874672518206976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830113.686140:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518206976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830113.686149:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830113.686154:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830113.686158:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518206976:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830113.686164:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518207104:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830113.686186:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830113.686197:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830113/real 1787830113] req@ffff8b2e8803a580 x1874672518207104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830113.686209:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518207104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830113.686216:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830113.686219:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830113.686222:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518207104:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830113.686228:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518207232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830113.686246:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830113.686258:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830113/real 1787830113] req@ffff8b2e88038f00 x1874672518207232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830113.686269:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518207232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830113.686276:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830113.686278:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830113.686282:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518207232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830113.686287:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518207360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830113.686304:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830113.686314:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830113/real 1787830113] req@ffff8b2e8803a1c0 x1874672518207360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830113.686337:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518207360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830113.686344:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830113.686346:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830113.686350:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518207360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830113.686355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518207488:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830113.686372:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830113.686383:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830113/real 1787830113] req@ffff8b2e8803b0c0 x1874672518207488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830113.686393:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518207488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830113.686401:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830113.686404:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830113.686407:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518207488:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830113.686412:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518207616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830113.686430:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830113.686440:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830113/real 1787830113] req@ffff8b2e88039a40 x1874672518207616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830113.686457:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518207616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830113.686464:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830113.686467:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830113.686470:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518207616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830113.686475:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518207744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830113.686492:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830113.686501:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830113/real 1787830113] req@ffff8b2e88039e00 x1874672518207744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830113.686511:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518207744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830138 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830113.686518:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830113.686521:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830113.686524:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518207744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830114.005149:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830114.516776:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830114.517103:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830114.708835:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830114.709609:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830115.029456:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830115.540783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830115.541034:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830115.732843:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830115.733508:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830116.052784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830116.564765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830116.564829:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830116.756761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830116.756835:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830117.077168:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830117.588789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830117.588834:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830117.780810:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830117.780810:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830118.100783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830118.613048:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830118.613103:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830118.804793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830118.804813:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830118.804819:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830118.804823:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830118.804832:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3125 00000100:00080000:3.0:1787830118.804843:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830118.804851:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830118.804887:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518207872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830118.804981:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830118.804985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830118.804987:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830118.804991:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3125 00000100:00080000:3.0:1787830118.804995:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830118.804999:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830118.805009:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830118.805031:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518208000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830118.805034:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518207872:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830118.805050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830118.805053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830118.805056:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830118.805060:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3125 00000100:00080000:3.0:1787830118.805064:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830118.805068:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830118.805073:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830118.805084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518208128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830118.805093:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787830118.805099:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830118.805099:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830118.805115:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830118.805120:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3125 00000100:00080000:3.0:1787830118.805124:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830118.805128:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830118.805132:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830118.805144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518208256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830118.805149:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830118/real 1787830118] req@ffff8b2e8803b480 x1874672518207872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830118.805168:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830118.805171:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518207872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830118.805173:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830118.805178:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830118.805187:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3125 00000100:00080000:3.0:1787830118.805194:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830118.805200:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:1.0:1787830118.805202:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787830118.805207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787830118.805209:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830118.805216:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518207872:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830118.805225:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518208384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830118.805238:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830118.805240:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518208000:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830118.805243:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830118.805248:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830118.805252:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3125 00000100:00080000:3.0:1787830118.805256:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830118.805260:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830118.805265:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830118.805276:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518208512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830118.805291:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830118.805302:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830118.805306:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:1.0:1787830118.805309:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830118/real 1787830118] req@ffff8b2e880392c0 x1874672518208000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830118.805309:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830118.805317:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3125 00000100:00080000:3.0:1787830118.805324:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830118.805330:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830118.805334:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518208000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787830118.805336:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830118.805337:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830118.805351:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787830118.805356:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518208640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830118.805360:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830118.805368:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518208000:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830118.805388:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518208128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830118.805409:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830118.805424:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830118/real 1787830118] req@ffff8b2e88039680 x1874672518208128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830118.805435:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518208128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830118.805441:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830118.805444:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830118.805447:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518208128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830118.805453:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518208256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830118.805469:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830118.805479:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830118/real 1787830118] req@ffff8b2e880383c0 x1874672518208256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830118.805487:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518208256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830118.805498:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830118.805511:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830118.805513:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518208256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830118.805527:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518208384:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830118.805543:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830118.805554:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830118/real 1787830118] req@ffff8b2e88039e00 x1874672518208384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830118.805564:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518208384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830118.805573:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830118.805575:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830118.805578:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518208384:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830118.805583:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518208512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830118.805715:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830118.805728:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830118/real 1787830118] req@ffff8b2e88039a40 x1874672518208512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830118.805752:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518208512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830118.805760:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830118.805763:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830118.805766:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518208512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830118.805896:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518208640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830118.805932:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830118.805944:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830118/real 1787830118] req@ffff8b2e8803b0c0 x1874672518208640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830118.805955:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518208640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830143 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830118.805963:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830118.805966:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830118.805970:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518208640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830119.124799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830119.636784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830119.636848:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830119.829445:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830119.830386:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830120.148801:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830120.660781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830120.660903:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830120.852785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830120.852828:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830121.172793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830121.684976:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830121.685043:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830121.876779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830121.876875:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830122.198467:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830122.708762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830122.708834:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830122.900761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830122.901035:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830123.220839:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830123.732801:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830123.735478:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830123.924763:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830123.924763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830123.924853:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830123.924860:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830123.924864:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830123.924873:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3130 00000100:00080000:3.0:1787830123.924883:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830123.924890:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830123.924929:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518208768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830123.924954:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830123.924956:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830123.924958:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830123.924961:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3130 00000100:00080000:3.0:1787830123.924964:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830123.924967:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830123.924975:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830123.924987:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518208896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830123.924997:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830123.925000:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787830123.925004:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518208768:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830123.925004:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830123.925013:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3130 00000100:00080000:3.0:1787830123.925017:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830123.925021:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830123.925026:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830123.925045:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518209024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830123.925054:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830123.925057:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830123.925059:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830123.925062:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3130 00000100:00080000:3.0:1787830123.925065:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830123.925068:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830123.925073:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830123.925085:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518209152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830123.925093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830123.925096:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830123.925098:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830123.925103:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518208896:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830123.925104:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3130 00000100:00080000:3.0:1787830123.925111:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830123.925116:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830123.925121:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830123.925135:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518209024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830123.925136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518209280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830123.925153:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518209152:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830123.925153:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830123.925159:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830123.925161:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830123.925166:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3130 00000100:00080000:3.0:1787830123.925170:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830123.925173:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518209280:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830123.925177:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830123.925183:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830123.925196:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518209408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830123.925206:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830123.925209:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830123.925211:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830123.925215:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3130 00000100:00100000:0.0:1787830123.925216:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518209408:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830123.925219:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830123.925225:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830123.925230:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830123.925242:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518209536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830123.925282:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518209536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830124.244785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830124.757576:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830124.759435:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830124.948796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830124.948797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830125.269624:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830125.780791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830125.780977:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830125.972786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830125.972958:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830126.292824:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830126.804772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830126.805065:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830126.997357:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:1.0:1787830126.997373:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787830126.997428:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787830126.997442:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830126.997474:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830126.997483:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830126.997491:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830126.997499:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830126.997506:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830126.997513:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787830126.998337:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830123/real 1787830126] req@ffff8b2e88039e00 x1874672518209536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830126.998369:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518209536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830126.998371:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830126.998390:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830126.998396:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830126.998401:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518209536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830126.998410:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830123/real 1787830126] req@ffff8b2e88039a40 x1874672518209408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830126.998421:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518209408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830126.998429:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830126.998432:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830126.998435:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518209408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830126.998441:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830123/real 1787830126] req@ffff8b2e8803b0c0 x1874672518209280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830126.998453:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518209280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830126.998462:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830126.998465:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830126.998469:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518209280:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830126.998475:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830123/real 1787830126] req@ffff8b2e88038f00 x1874672518208896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830126.998486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518208896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830126.998494:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830126.998498:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830126.998501:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518208896:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830126.998508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830123/real 1787830126] req@ffff8b2e8803a580 x1874672518209024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830126.998519:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518209024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830126.998527:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830126.998530:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830126.998533:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518209024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830126.998539:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830123/real 1787830126] req@ffff8b2e8803b840 x1874672518209152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830126.998561:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518209152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830126.998568:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830126.998571:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830126.998574:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518209152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830126.998581:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830123/real 1787830126] req@ffff8b2e8803a1c0 x1874672518208768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830126.998592:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518208768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830148 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830126.998599:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830126.998602:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830126.998604:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518208768:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830127.317105:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830127.828958:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830127.830741:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830128.021313:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830128.021326:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830128.340873:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830128.854236:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830128.854723:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830129.044868:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830129.044879:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787830129.044882:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830129.044883:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830129.044884:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830129.044897:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3135 00000100:00080000:0.0:1787830129.044910:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830129.044918:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830129.044959:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518209664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830129.044983:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830129.044987:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830129.044989:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830129.044993:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3135 00000100:00080000:0.0:1787830129.044996:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830129.045000:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830129.045009:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830129.045021:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518209792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830129.045030:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830129.045033:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830129.045035:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830129.045038:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3135 00000100:00080000:0.0:1787830129.045041:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830129.045045:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830129.045049:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830129.045058:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518209920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830129.045066:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830129.045069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830129.045071:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830129.045074:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3135 00000100:00080000:0.0:1787830129.045077:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830129.045080:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830129.045084:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830129.045092:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518210048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830129.045100:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830129.045102:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830129.045104:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830129.045107:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3135 00000100:00080000:0.0:1787830129.045110:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830129.045113:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830129.045117:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830129.045186:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518210176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830129.045197:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830129.045200:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830129.045202:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830129.045206:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3135 00000100:00080000:0.0:1787830129.045221:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830129.045226:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830129.045231:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830129.045244:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518210304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830129.045251:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518209664:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787830129.045257:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830129.045261:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830129.045263:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830129.045267:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3135 00000100:00080000:0.0:1787830129.045271:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830129.045275:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830129.045280:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830129.045296:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518210432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830129.045311:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830129.045352:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830129/real 1787830129] req@ffff8b2e880421c0 x1874672518209664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830129.045371:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518209664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830129.045381:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830129.045386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830129.045389:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518209664:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830129.045397:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518209792:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830129.045431:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830129.045445:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830129/real 1787830129] req@ffff8b2e88041a40 x1874672518209792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830129.045456:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518209792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830129.045463:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830129.045466:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830129.045469:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518209792:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830129.045475:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518209920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830129.045495:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830129.045505:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830129/real 1787830129] req@ffff8b2e88040f00 x1874672518209920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830129.045516:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518209920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830129.045522:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830129.045525:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830129.045528:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518209920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830129.045533:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518210048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830129.045550:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830129.045559:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830129/real 1787830129] req@ffff8b2e88042940 x1874672518210048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830129.045570:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518210048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830129.045577:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830129.045580:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830129.045582:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518210048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830129.045587:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518210176:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830129.045605:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830129.045615:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830129/real 1787830129] req@ffff8b2e88043480 x1874672518210176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830129.045625:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518210176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830129.045633:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830129.045636:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830129.045638:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518210176:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830129.045684:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518210304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830129.045704:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830129.045714:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830129/real 1787830129] req@ffff8b2e880430c0 x1874672518210304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830129.045724:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518210304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830129.045731:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830129.045734:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830129.045737:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518210304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830129.045742:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518210432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830129.045758:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830129.045768:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830129/real 1787830129] req@ffff8b2e88040000 x1874672518210432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830129.045779:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518210432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830154 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830129.045792:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830129.045795:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830129.045798:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518210432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830129.364797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830129.876859:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830129.877971:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830130.068786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830130.068786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830130.388792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830130.900725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830130.900791:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830131.093015:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830131.093015:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830131.412792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830131.924945:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830131.925174:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830132.116786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830132.116792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830132.436824:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830132.948781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830132.948821:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830133.140785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830133.141085:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830133.461335:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830133.972792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830133.977144:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830134.164812:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830134.164824:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830134.164829:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830134.164839:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3140 00000100:00080000:1.0:1787830134.164850:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830134.164858:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830134.164892:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518210560/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830134.164916:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830134.164920:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830134.164922:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830134.164926:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3140 00000100:00080000:1.0:1787830134.164929:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830134.164933:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830134.164943:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830134.164958:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518210688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830134.164970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830134.164973:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830134.164974:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830134.164977:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3140 00000100:00080000:1.0:1787830134.164980:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830134.164983:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830134.164987:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830134.164996:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518210560:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830134.164999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518210816/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830134.165002:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830134.165013:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830134.165015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830134.165017:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830134.165020:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3140 00000100:00080000:1.0:1787830134.165023:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830134.165025:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830134.165027:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000008:00100000:1.0:1787830134.165029:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830134.165039:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518210944/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830134.165045:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830134.165048:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830134.165050:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830134.165052:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3140 00000100:00080000:1.0:1787830134.165056:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830134.165059:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830134.165062:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830134.165064:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830134.165070:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518211072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830134.165077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830134.165079:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830134.165080:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830134.165083:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3140 00000100:00080000:1.0:1787830134.165085:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830134.165086:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830134.165089:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830134.165095:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518211200/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830134.165099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830134.165101:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830134.165103:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830134.165107:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3140 00000100:00100000:0.0:1787830134.165108:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830134/real 1787830134] req@ffff8b2e88076580 x1874672518210560/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830134.165109:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830134.165113:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830134.165118:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830134.165128:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518211328/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830134.165135:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518210560/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830134.165159:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830134.165164:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830134.165168:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518210560:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830134.165185:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518210688:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830134.165209:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830134.165222:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830134/real 1787830134] req@ffff8b2e88075e00 x1874672518210688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830134.165234:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518210688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830134.165242:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830134.165245:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830134.165249:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518210688:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830134.165271:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518210816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830134.165292:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830134.165303:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830134/real 1787830134] req@ffff8b2e88074b40 x1874672518210816/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830134.165315:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518210816/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830134.165323:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830134.165326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830134.165329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518210816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830134.165335:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518210944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830134.165353:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830134.165364:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830134/real 1787830134] req@ffff8b2e88074000 x1874672518210944/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830134.165388:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518210944/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830134.165396:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830134.165399:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830134.165402:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518210944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830134.165408:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518211072:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830134.165426:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830134.165437:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830134/real 1787830134] req@ffff8b2e880761c0 x1874672518211072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830134.165448:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518211072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830134.165456:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830134.165459:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830134.165463:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518211072:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830134.165469:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518211200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830134.165488:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830134.165499:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830134/real 1787830134] req@ffff8b2e880770c0 x1874672518211200/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830134.165512:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518211200/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830134.165520:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830134.165522:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830134.165525:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518211200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830134.165531:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518211328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830134.165552:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830134.165563:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830134/real 1787830134] req@ffff8b2e88077840 x1874672518211328/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830134.165574:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518211328/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830159 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830134.165581:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830134.165584:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830134.165586:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518211328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830134.484808:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830134.996992:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830134.998261:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830135.190180:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830135.190180:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830135.508796:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830136.020782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830136.020858:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830136.212765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830136.212765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830136.532762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830137.044866:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830137.045225:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830137.236735:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830137.236765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830137.556774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830138.069062:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830138.069254:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830138.260922:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830138.261231:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830138.581755:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830139.092756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830139.092881:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830139.284793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830139.284793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830139.285068:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830139.285076:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830139.285079:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830139.285088:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3145 00000100:00080000:0.0:1787830139.285098:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830139.285105:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830139.285155:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518211456/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830139.285182:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830139.285186:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830139.285188:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830139.285191:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3145 00000100:00080000:0.0:1787830139.285194:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830139.285199:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830139.285208:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830139.285222:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518211456:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830139.285226:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518211584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830139.285238:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830139.285242:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830139.285244:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830139.285247:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3145 00000100:00080000:0.0:1787830139.285251:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830139.285255:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830139.285260:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830139.285271:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518211712/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830139.285281:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787830139.285285:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830139.285286:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830139.285290:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830139.285295:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3145 00000100:00080000:0.0:1787830139.285299:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830139.285303:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830139.285308:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830139.285319:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518211840/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830139.285328:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830139.285329:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830139/real 1787830139] req@ffff8b2e880403c0 x1874672518211456/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830139.285336:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830139.285341:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830139.285355:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518211456/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830139.285361:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3145 00000100:00080000:0.0:1787830139.285369:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830139.285372:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830139.285376:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:1.0:1787830139.285381:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:0.0:1787830139.285385:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830139.285388:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518211456:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830139.285399:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518211968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830139.285410:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830139.285416:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787830139.285416:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518211584:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787830139.285420:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830139.285427:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3145 00000100:00080000:0.0:1787830139.285432:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830139.285436:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830139.285441:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787830139.285446:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830139.285455:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518212096/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830139.285466:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830139.285469:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830139.285471:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830139.285478:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3145 00000100:00100000:1.0:1787830139.285478:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830139/real 1787830139] req@ffff8b2e88040000 x1874672518211584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830139.285485:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830139.285490:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830139.285496:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830139.285499:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518211584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830139.285510:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787830139.285511:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518212224/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830139.285521:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830139.285527:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518211584:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830139.285539:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518211712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830139.285570:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830139.285579:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830139/real 1787830139] req@ffff8b2e880430c0 x1874672518211712/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830139.285588:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518211712/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830139.285595:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830139.285598:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830139.285601:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518211712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830139.285606:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518211840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830139.285624:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830139.285634:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830139/real 1787830139] req@ffff8b2e88043480 x1874672518211840/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830139.285671:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518211840/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830139.285680:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830139.285684:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830139.285687:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518211840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830139.285698:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518211968:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830139.285717:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830139.285727:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830139/real 1787830139] req@ffff8b2e88042940 x1874672518211968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830139.285738:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518211968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830139.285746:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830139.285749:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830139.285752:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518211968:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830139.285768:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518212096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830139.285787:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830139.285798:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830139/real 1787830139] req@ffff8b2e88040f00 x1874672518212096/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830139.285809:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518212096/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830139.285817:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830139.285820:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830139.285823:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518212096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830139.285828:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518212224:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830139.285845:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830139.285856:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830139/real 1787830139] req@ffff8b2e88041a40 x1874672518212224/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830139.285866:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518212224/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830164 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830139.285878:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830139.285881:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830139.285884:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518212224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830139.604955:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830140.116823:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830140.117369:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830140.308995:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830140.309284:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830140.628813:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830141.140777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830141.141627:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830141.332783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830141.332783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830141.652797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830142.164792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830142.164817:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830142.356846:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830142.356976:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830142.677628:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830143.189196:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830143.189253:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830143.380780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830143.381109:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830143.700892:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830144.212777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830144.212831:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830144.404746:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830144.404780:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830144.404786:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830144.404790:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830144.404806:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3150 00000100:00080000:0.0:1787830144.404816:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830144.404824:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830144.404895:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518212352/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830144.404931:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830144.404934:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830144.404936:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830144.404939:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3150 00000100:00080000:0.0:1787830144.404942:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830144.404945:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830144.404957:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830144.404970:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518212480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830144.404979:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830144.404988:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830144.404990:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830144.404993:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3150 00000100:00080000:0.0:1787830144.404997:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830144.405001:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830144.405005:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830144.405022:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518212608/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830144.405031:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830144.405034:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830144.405036:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830144.405039:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3150 00000100:00080000:0.0:1787830144.405043:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830144.405054:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830144.405058:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830144.405069:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518212736/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830144.405084:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830144.405088:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830144.405089:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830144.405093:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3150 00000100:00080000:0.0:1787830144.405096:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830144.405115:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830144.405121:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830144.405131:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518212864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830144.405147:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830144.405151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830144.405155:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830144.405155:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518212352:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787830144.405161:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3150 00000100:00080000:0.0:1787830144.405166:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830144.405176:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830144.405181:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830144.405194:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518212992/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830144.405209:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830144.405218:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787830144.405222:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830144.405223:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830144.405230:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3150 00000100:00080000:0.0:1787830144.405236:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830144.405248:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830144.405253:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830144.405267:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518213120/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830144.405274:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830144/real 1787830144] req@ffff8b2e880421c0 x1874672518212352/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830144.405296:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518212352/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830144.405307:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830144.405312:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830144.405316:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518212352:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830144.405323:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518212480:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830144.405345:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830144.405358:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830144/real 1787830144] req@ffff8b2e88040b40 x1874672518212480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830144.405370:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518212480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830144.405384:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830144.405387:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830144.405390:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518212480:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830144.405396:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518212608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830144.405417:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830144.405428:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830144/real 1787830144] req@ffff8b2e880412c0 x1874672518212608/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830144.405439:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518212608/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830144.405446:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830144.405449:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830144.405452:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518212608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830144.405458:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518212736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830144.405476:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830144.405490:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830144/real 1787830144] req@ffff8b2e88043840 x1874672518212736/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830144.405498:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830144.405504:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518212736/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830144.405513:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830144.405517:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830144.405520:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518212736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830144.405545:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518212864:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830144.405564:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830144.405574:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830144/real 1787830144] req@ffff8b2e88040780 x1874672518212864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830144.405586:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518212864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830144.405594:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830144.405597:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830144.405600:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518212864:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830144.405605:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518212992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830144.405623:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830144.405633:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830144/real 1787830144] req@ffff8b2e88042d00 x1874672518212992/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830144.405644:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518212992/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830144.405674:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830144.405678:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830144.405681:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518212992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830144.405687:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518213120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830144.405706:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830144.405717:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830144/real 1787830144] req@ffff8b2e88042580 x1874672518213120/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830144.405728:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518213120/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830169 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830144.405736:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830144.405739:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830144.405742:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518213120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830144.724788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830145.236775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830145.236867:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830145.428772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830145.428772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830145.748784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830146.260771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830146.260927:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830146.452850:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830146.453056:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830146.772862:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830147.284777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830147.284972:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830147.476778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830147.476779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830147.796774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830148.308750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830148.308971:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830148.500778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830148.500778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830148.820788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830149.332890:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830149.333056:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830149.524754:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830149.524757:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830149.524767:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830149.524772:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830149.524781:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3155 00000100:00080000:3.0:1787830149.524791:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830149.524797:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830149.524831:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518213248/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830149.524839:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830149.524858:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830149.524862:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830149.524865:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830149.524868:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3155 00000100:00080000:3.0:1787830149.524871:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830149.524874:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830149.524882:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830149.524891:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518213376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830149.524897:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830149.524899:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830149.524900:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830149.524903:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3155 00000100:00080000:3.0:1787830149.524904:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830149.524906:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830149.524909:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830149.524914:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518213504/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830149.524919:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830149.524921:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830149.524922:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830149.524924:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3155 00000100:00080000:3.0:1787830149.524925:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830149.524927:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830149.524929:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830149.524934:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518213632/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830149.524941:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830149.524943:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830149.524945:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830149.524948:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3155 00000100:00080000:3.0:1787830149.524950:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830149.524954:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830149.524957:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830149.524970:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518213760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830149.524979:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830149.524981:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830149.524983:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830149.524987:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3155 00000100:00080000:3.0:1787830149.524989:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830149.524997:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830149.525001:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830149.525008:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518213888/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830149.525013:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830149.525015:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830149.525016:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830149.525017:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3155 00000100:00080000:3.0:1787830149.525019:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830149.525021:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830149.525023:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830149.525028:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518214016/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830149.525605:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518213248:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830149.525683:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830149.525728:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830149/real 1787830149] req@ffff8b2e880383c0 x1874672518213248/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830149.525744:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518213248/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830149.525754:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830149.525758:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830149.525762:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518213248:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830149.525769:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518213376:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830149.525792:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830149.525804:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830149/real 1787830149] req@ffff8b2e88039680 x1874672518213376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830149.525816:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518213376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830149.525824:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830149.525827:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830149.525830:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518213376:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830149.525836:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518213504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830149.525855:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830149.525865:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830149/real 1787830149] req@ffff8b2e880392c0 x1874672518213504/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830149.525877:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518213504/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830149.525884:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830149.525887:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830149.525890:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518213504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830149.525895:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518213632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830149.525913:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830149.525923:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830149/real 1787830149] req@ffff8b2e8803b480 x1874672518213632/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830149.525934:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518213632/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830149.525942:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830149.525944:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830149.525947:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518213632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830149.525953:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518213760:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830149.525969:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830149.525980:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830149/real 1787830149] req@ffff8b2e8803a1c0 x1874672518213760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830149.525991:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518213760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830149.525999:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830149.526002:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830149.526005:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518213760:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830149.526011:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518213888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830149.526028:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830149.526039:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830149/real 1787830149] req@ffff8b2e8803b840 x1874672518213888/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830149.526049:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518213888/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830149.526057:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830149.526060:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830149.526063:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518213888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830149.526080:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518214016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830149.526100:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830149.526110:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830149/real 1787830149] req@ffff8b2e8803a580 x1874672518214016/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830149.526121:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518214016/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830174 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830149.526129:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830149.526131:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830149.526134:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518214016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830149.844797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830150.357121:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830150.357559:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830150.549070:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830150.549486:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830150.868783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830151.380776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830151.381368:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830151.572775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830151.572887:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830151.893196:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830152.404875:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830152.405175:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830152.596778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830152.596778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830152.918358:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830153.428859:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830153.429105:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830153.620732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830153.620798:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830153.940770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830154.452791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830154.452903:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830154.645533:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830154.645800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830154.647190:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830154.647200:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830154.647204:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830154.647214:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3161 00000100:00080000:0.0:1787830154.647224:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830154.647231:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830154.647270:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518214144/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830154.647345:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518214144:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830154.647416:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830154.647447:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830154.647451:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830154.647453:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830154.647457:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3161 00000100:00080000:0.0:1787830154.647461:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830154.647466:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830154.647475:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830154.647490:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518214272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830154.648311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830154/real 1787830154] req@ffff8b2e88041e00 x1874672518214144/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830154.648333:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518214144/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830154.648343:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830154.648358:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830154.648363:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518214144:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830154.648391:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518214272:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830154.648423:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830154.648435:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830154/real 1787830154] req@ffff8b2e88042580 x1874672518214272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830154.648447:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518214272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830154.648456:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830154.648459:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830154.648462:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518214272:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787830154.648714:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830154.648891:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830154.648894:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830154.648901:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3161 00000100:00080000:0.0:1787830154.648907:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830154.648912:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830154.648921:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830154.648942:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518214400/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830154.648954:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830154.648957:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830154.648959:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830154.648962:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3161 00000100:00080000:0.0:1787830154.648965:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830154.648968:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830154.648973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830154.648983:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518214528/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830154.648991:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830154.648995:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830154.649070:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830154.649075:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3161 00000100:00080000:0.0:1787830154.649079:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830154.649148:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830154.649154:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830154.649155:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518214400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830154.649171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518214656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830154.649191:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830154.649206:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830154/real 1787830154] req@ffff8b2e88042d00 x1874672518214400/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830154.649211:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830154.649219:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830154.649222:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830154.649231:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3161 00000100:00100000:2.0:1787830154.649236:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518214400/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830154.649237:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830154.649246:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787830154.649249:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787830154.649253:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787830154.649256:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830154.649260:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518214400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830154.649266:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518214784/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830154.649270:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518214528:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787830154.649281:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830154.649284:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830154.649285:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830154.649290:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3161 00000100:00080000:0.0:1787830154.649295:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830154.649298:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830154.649303:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787830154.649313:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830154.649315:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518214912/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830154.649329:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830154/real 1787830154] req@ffff8b2e88040780 x1874672518214528/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830154.649342:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518214528/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830154.649350:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830154.649354:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830154.649356:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518214528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830154.649371:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518214656:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830154.649390:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830154.649401:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830154/real 1787830154] req@ffff8b2e88043840 x1874672518214656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830154.649412:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518214656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830154.649425:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830154.649428:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830154.649431:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518214656:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830154.649436:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518214784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830154.649452:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830154.649462:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830154/real 1787830154] req@ffff8b2e880412c0 x1874672518214784/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830154.649471:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518214784/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830154.649477:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830154.649480:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830154.649483:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518214784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830154.649487:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518214912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830154.649502:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830154.649511:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830154/real 1787830154] req@ffff8b2e88040b40 x1874672518214912/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830154.649521:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518214912/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830179 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830154.649528:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830154.649531:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830154.649534:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518214912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830154.966469:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830155.476901:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830155.477290:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830155.668764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830155.669309:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830155.989016:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830156.500835:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830156.500854:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830156.692899:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830156.693004:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830157.012815:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830157.524758:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830157.525038:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830157.716748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830157.716748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830158.037082:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830158.548902:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830158.550509:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830158.740787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830158.742540:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830159.060906:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830159.572771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830159.572812:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830159.764786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830159.764787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830159.765995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830159.766012:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830159.766016:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830159.766024:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3166 00000100:00080000:3.0:1787830159.766035:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830159.766041:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830159.766076:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518215040/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830159.766143:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518215040:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830159.766195:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830159.766223:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830159.766227:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830159.766229:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830159.766234:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3166 00000100:00080000:3.0:1787830159.766238:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830159.766242:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830159.766252:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830159.766265:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518215168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830159.766386:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830159/real 1787830159] req@ffff8b2e88038f00 x1874672518215040/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830159.766404:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518215040/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830159.766414:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830159.766417:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830159.766421:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518215040:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830159.766457:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518215168:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830159.766478:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830159.766489:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830159/real 1787830159] req@ffff8b2e8803b0c0 x1874672518215168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830159.766501:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518215168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830159.766510:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830159.766513:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830159.766516:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518215168:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830159.767356:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830159.767362:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830159.767365:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830159.767371:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3166 00000100:00080000:3.0:1787830159.767376:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830159.767380:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830159.767386:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830159.767405:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518215296/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830159.767416:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830159.767419:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830159.767421:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830159.767424:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3166 00000100:00080000:3.0:1787830159.767427:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830159.767448:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830159.767453:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830159.767463:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518215424/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830159.767472:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830159.767475:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830159.767477:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830159.767481:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3166 00000100:00080000:3.0:1787830159.767484:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830159.767487:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830159.767490:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830159.767498:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518215552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830159.767506:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830159.767509:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830159.767511:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830159.767514:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3166 00000100:00080000:3.0:1787830159.767517:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830159.767521:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830159.767524:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830159.767542:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518215680/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830159.767550:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830159.767553:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830159.767554:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830159.767558:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3166 00000100:00080000:3.0:1787830159.767562:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830159.767565:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830159.767568:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830159.767578:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518215808/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830159.767670:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518215296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830159.767702:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830159.767717:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830159/real 1787830159] req@ffff8b2e88039a40 x1874672518215296/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830159.767729:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518215296/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830159.767738:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830159.767741:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830159.767744:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518215296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830159.767750:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518215424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830159.767766:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830159.767775:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830159/real 1787830159] req@ffff8b2e88039e00 x1874672518215424/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830159.767786:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518215424/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830159.767793:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830159.767796:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830159.767800:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518215424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830159.767805:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518215552:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830159.767819:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830159.767828:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830159/real 1787830159] req@ffff8b2e8803b0c0 x1874672518215552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830159.767839:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518215552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830159.767848:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830159.767850:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830159.767853:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518215552:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830159.767858:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518215680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830159.767874:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830159.767883:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830159/real 1787830159] req@ffff8b2e88038f00 x1874672518215680/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830159.767894:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518215680/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830159.767901:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830159.767904:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830159.767907:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518215680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830159.767912:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518215808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830159.767926:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830159.767935:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830159/real 1787830159] req@ffff8b2e8803a580 x1874672518215808/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830159.767945:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518215808/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830184 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830159.768181:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830159.768185:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830159.768188:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518215808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830160.084782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830160.596731:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830160.596887:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830160.789312:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830160.789490:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830161.108789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830161.620916:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830161.621547:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830161.812796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830161.813003:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830162.132780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830162.644894:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830162.645318:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830162.836794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830162.837316:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830163.157218:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830163.668791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830163.669080:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830163.860785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830163.860911:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830164.180789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830164.692982:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830164.694086:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:3.0:1787830164.884772:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830164.884780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830164.884784:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830164.884792:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830164.884804:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3171 00000100:00080000:3.0:1787830164.884818:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830164.884828:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830164.884875:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518215936/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830164.884905:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830164.884909:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830164.884911:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830164.884916:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3171 00000100:00080000:3.0:1787830164.884920:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830164.884925:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830164.884937:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830164.884953:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518216064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830164.884975:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830164.884979:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830164.884982:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830164.884986:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3171 00000100:00080000:3.0:1787830164.884990:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830164.884995:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830164.885001:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830164.885014:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518216192/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830164.885024:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830164.885027:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830164.885029:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830164.885035:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3171 00000100:00080000:3.0:1787830164.885039:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830164.885046:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830164.885052:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518215936:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787830164.885053:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830164.885078:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518216320/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830164.885090:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830164.885092:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830164.885099:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830164.885102:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830164.885108:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3171 00000100:00080000:3.0:1787830164.885126:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830164.885131:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830164.885136:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787830164.885144:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830164.885155:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518216448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830164.885168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830164.885172:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830164.885174:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830164.885179:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3171 00000100:00080000:3.0:1787830164.885183:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830164.885189:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830164.885194:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830164.885201:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830164/real 1787830164] req@ffff8b2e8803b840 x1874672518215936/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830164.885210:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518216576/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830164.885232:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518215936/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830164.885243:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830164.885257:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830164.885260:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830164.885263:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830164.885268:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3171 00000100:00080000:2.0:1787830164.885274:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787830164.885277:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830164.885283:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518215936:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830164.885285:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830164.885301:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830164.885305:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518216064:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830164.885323:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518216704/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830164.885354:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830164.885383:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830164/real 1787830164] req@ffff8b2e8803a1c0 x1874672518216064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830164.885395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518216064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830164.885404:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830164.885407:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830164.885410:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518216064:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830164.885417:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518216192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830164.885436:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830164.885446:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830164/real 1787830164] req@ffff8b2e8803b480 x1874672518216192/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830164.885457:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518216192/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830164.885465:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830164.885469:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830164.885472:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518216192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830164.885496:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518216320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830164.885513:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830164.885523:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830164/real 1787830164] req@ffff8b2e880392c0 x1874672518216320/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830164.885541:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518216320/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830164.885550:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830164.885553:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830164.885565:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518216320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830164.885572:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518216448:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830164.885590:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830164.885601:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830164/real 1787830164] req@ffff8b2e88039680 x1874672518216448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830164.885612:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518216448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830164.885621:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830164.885624:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830164.885627:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518216448:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830164.885633:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518216576:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830164.885674:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830164.885685:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830164/real 1787830164] req@ffff8b2e880383c0 x1874672518216576/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830164.885697:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518216576/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830164.885710:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830164.885713:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830164.885717:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518216576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830164.885723:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518216704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830164.885739:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830164.885749:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830164/real 1787830164] req@ffff8b2e8803a580 x1874672518216704/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830164.885759:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518216704/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830189 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830164.885767:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830164.885771:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830164.885774:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518216704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830165.205148:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830165.716819:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830165.716841:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830165.908792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830165.908792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830166.229138:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830166.741039:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830166.742012:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830166.932770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830166.932770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830167.254081:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830167.764783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830167.765147:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830167.956770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830167.956770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830168.276793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830168.788811:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830168.788844:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830168.981071:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830168.981619:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830169.300738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830169.812782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830169.812916:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830170.004788:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830170.004790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830170.004794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787830170.004800:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830170.004818:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830170.004830:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3176 00000100:00080000:2.0:1787830170.004850:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830170.004866:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830170.004910:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518216832/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830170.004952:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830170.004959:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830170.004960:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830170.004965:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3176 00000100:00080000:2.0:1787830170.004968:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830170.004972:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830170.004983:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830170.004995:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518216960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830170.005004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830170.005008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830170.005009:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830170.005012:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3176 00000100:00080000:2.0:1787830170.005015:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830170.005019:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830170.005024:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830170.005034:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518217088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830170.005044:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830170.005047:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830170.005048:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830170.005052:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3176 00000100:00080000:2.0:1787830170.005055:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830170.005058:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830170.005063:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830170.005073:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518217216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830170.005082:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830170.005085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830170.005087:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830170.005091:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3176 00000100:00080000:2.0:1787830170.005095:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830170.005099:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830170.005103:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830170.005113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518217344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830170.005122:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830170.005125:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830170.005127:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830170.005131:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3176 00000100:00080000:2.0:1787830170.005134:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830170.005137:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830170.005141:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830170.005150:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518217472/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830170.005159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830170.005163:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830170.005164:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830170.005167:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3176 00000100:00080000:2.0:1787830170.005171:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830170.005174:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830170.005178:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830170.005188:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518217600/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830170.005525:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518216832:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830170.005591:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830170.005639:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830170/real 1787830170] req@ffff8b2e8801ed00 x1874672518216832/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830170.005680:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518216832/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830170.005690:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830170.005695:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830170.005699:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518216832:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830170.005706:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518216960:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830170.005728:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830170.005741:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830170/real 1787830170] req@ffff8b2e8801c780 x1874672518216960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830170.005752:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518216960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830170.005760:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830170.005763:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830170.005767:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518216960:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830170.005773:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518217088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830170.005790:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830170.005800:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830170/real 1787830170] req@ffff8b2e8801f0c0 x1874672518217088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830170.005811:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518217088/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830170.005818:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830170.005821:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830170.005831:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518217088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830170.005838:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518217216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830170.005857:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830170.005867:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830170/real 1787830170] req@ffff8b2e8801d680 x1874672518217216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830170.005879:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518217216/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830170.005886:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830170.005890:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830170.005893:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518217216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830170.005900:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518217344:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830170.005917:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830170.005938:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830170/real 1787830170] req@ffff8b2e8801c3c0 x1874672518217344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830170.005949:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518217344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830170.005956:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830170.005959:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830170.005962:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518217344:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830170.005968:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518217472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830170.005986:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830170.005996:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830170/real 1787830170] req@ffff8b2e8801cb40 x1874672518217472/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830170.006014:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518217472/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830170.006026:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830170.006029:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830170.006032:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518217472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830170.006038:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518217600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830170.006056:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830170.006067:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830170/real 1787830170] req@ffff8b2e8801de00 x1874672518217600/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830170.006076:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518217600/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830195 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830170.006083:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830170.006086:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830170.006089:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518217600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830170.324800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830170.836775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830170.836883:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830171.028791:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830171.029341:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830171.349187:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830171.860785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830171.861249:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830172.052772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830172.052772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830172.372764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830172.884769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830172.885581:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830173.076984:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830173.077175:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830173.396822:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830173.908794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830173.909023:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830174.100719:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830174.100719:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830174.420888:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830174.933157:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830174.933521:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830175.124764:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830175.124766:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830175.124767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830175.124777:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830175.124782:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830175.124792:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3181 00000100:00080000:0.0:1787830175.124802:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830175.124809:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830175.124847:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518217728/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830175.124874:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830175.124877:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830175.124878:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830175.124881:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3181 00000100:00080000:0.0:1787830175.124884:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830175.124887:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830175.124907:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830175.124922:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518217856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830175.124925:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518217728:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787830175.124940:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830175.124944:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830175.124946:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830175.124950:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3181 00000100:00080000:0.0:1787830175.124954:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830175.124958:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830175.124963:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830175.124975:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518217984/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830175.124987:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830175.124995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830175.124999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830175.125001:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830175.125005:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3181 00000100:00080000:0.0:1787830175.125010:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830175.125014:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830175.125019:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830175.125032:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518218112/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830175.125036:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830175/real 1787830175] req@ffff8b2e880421c0 x1874672518217728/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830175.125050:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830175.125055:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830175.125057:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830175.125063:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3181 00000100:00100000:1.0:1787830175.125063:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518217728/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830175.125069:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830175.125077:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:1.0:1787830175.125081:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787830175.125083:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787830175.125087:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830175.125093:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518217728:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830175.125102:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518218240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830175.125112:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830175.125115:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:1.0:1787830175.125118:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518217856:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787830175.125119:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830175.125126:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3181 00000100:00080000:0.0:1787830175.125130:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830175.125133:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830175.125138:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787830175.125149:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830175.125150:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518218368/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830175.125164:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830175/real 1787830175] req@ffff8b2e88041a40 x1874672518217856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830175.125168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830175.125174:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830175.125176:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830175.125177:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518217856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830175.125185:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3181 00000100:00080000:1.0:1787830175.125189:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830175.125191:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830175.125194:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830175.125197:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518217856:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830175.125201:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518217984:192.168.203.101@tcp:8:kworker.0 00000100:00080000:0.0:1787830175.125208:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830175.125213:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787830175.125218:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830175.125225:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518218496/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830175.125232:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830175/real 1787830175] req@ffff8b2e88040f00 x1874672518217984/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830175.125249:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518217984/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830175.125257:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830175.125261:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830175.125264:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518217984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830175.125281:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518218112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830175.125296:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830175.125304:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830175/real 1787830175] req@ffff8b2e88042940 x1874672518218112/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830175.125316:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518218112/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830175.125323:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830175.125326:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830175.125329:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518218112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830175.125334:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518218240:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830175.125352:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830175.125368:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830175/real 1787830175] req@ffff8b2e88043480 x1874672518218240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830175.125379:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518218240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830175.125391:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830175.125394:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830175.125396:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518218240:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830175.125413:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518218368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830175.125432:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830175.125442:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830175/real 1787830175] req@ffff8b2e880430c0 x1874672518218368/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830175.125452:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518218368/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830175.125459:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830175.125461:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830175.125464:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518218368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830175.125469:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518218496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830175.125590:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830175.125602:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830175/real 1787830175] req@ffff8b2e88040000 x1874672518218496/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830175.125612:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518218496/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830200 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830175.125619:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830175.125622:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830175.125625:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518218496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830175.445065:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830175.956779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830175.956866:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830176.148813:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830176.149253:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830176.468912:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830176.980803:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830176.981943:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830177.172788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830177.172831:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830177.492798:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830178.004889:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830178.004907:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830178.196765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830178.196765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830178.516790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830179.028773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830179.028874:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830179.220750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830179.220753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830179.540772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830180.052771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830180.056446:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830180.244788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830180.244935:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830180.244941:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830180.244945:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830180.244954:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3186 00000100:00080000:3.0:1787830180.244964:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830180.244972:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830180.245016:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518218624/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830180.245104:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830180.245156:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518218624:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830180.245208:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830180.245238:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830180.245242:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830180.245244:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830180.245248:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3186 00000100:00080000:3.0:1787830180.245253:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830180.245257:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830180.245266:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830180.245279:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518218752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830180.245339:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830180/real 1787830180] req@ffff8b2e88038f00 x1874672518218624/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830180.245356:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518218624/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830180.245365:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830180.245369:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830180.245373:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518218624:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830180.245394:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518218752:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830180.245416:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830180.245429:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830180/real 1787830180] req@ffff8b2e8803b0c0 x1874672518218752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830180.245440:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518218752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830180.245448:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830180.245451:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830180.245454:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518218752:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830180.245499:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830180.245503:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830180.245505:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830180.245509:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3186 00000100:00080000:3.0:1787830180.245513:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830180.245517:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830180.245522:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830180.245534:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518218880/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830180.245544:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830180.245547:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830180.245548:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830180.245551:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3186 00000100:00080000:3.0:1787830180.245555:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830180.245558:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830180.245562:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830180.245572:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518219008/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830180.245580:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830180.245583:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830180.245585:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830180.245589:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3186 00000100:00080000:3.0:1787830180.245591:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830180.245595:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830180.245599:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830180.245609:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518219136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830180.245626:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830180.245629:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830180.245631:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830180.245634:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3186 00000100:00080000:3.0:1787830180.245637:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830180.245640:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830180.245673:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830180.245685:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518219264/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830180.245694:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830180.245697:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830180.245699:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830180.245702:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3186 00000100:00080000:3.0:1787830180.245704:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830180.245707:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830180.245711:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830180.245720:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518219392/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830180.245784:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518218880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830180.245807:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830180.245818:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830180/real 1787830180] req@ffff8b2e88039e00 x1874672518218880/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830180.245830:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518218880/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830180.245838:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830180.245841:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830180.245844:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518218880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830180.245849:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518219008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830180.245884:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830180.245894:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830180/real 1787830180] req@ffff8b2e88039a40 x1874672518219008/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830180.245906:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518219008/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830180.245913:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830180.245916:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830180.245920:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518219008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830180.245925:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518219136:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830180.245940:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830180.245948:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830180/real 1787830180] req@ffff8b2e8803b0c0 x1874672518219136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830180.245959:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518219136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830180.245967:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830180.245970:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830180.245973:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518219136:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830180.245978:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518219264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830180.245994:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830180.246003:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830180/real 1787830180] req@ffff8b2e88038f00 x1874672518219264/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830180.246013:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518219264/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830180.246020:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830180.246023:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830180.246026:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518219264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830180.246031:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518219392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830180.246048:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830180.246057:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830180/real 1787830180] req@ffff8b2e8803a580 x1874672518219392/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830180.246067:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518219392/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830205 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830180.246074:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830180.246078:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830180.246081:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518219392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830180.567486:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830181.076808:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830181.076955:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830181.268787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830181.268981:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830181.588919:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830182.100762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830182.100918:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830182.292795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830182.293271:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830182.613116:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830183.124722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830183.124846:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830183.316812:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830183.316947:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830183.636985:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830184.148793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830184.148852:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830184.340799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830184.343139:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830184.660786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830185.172869:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830185.172870:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:1.0:1787830185.364782:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830185.364793:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830185.364796:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830185.364806:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3191 00000100:00080000:1.0:1787830185.364817:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830185.364838:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830185.364875:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518219520/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830185.364910:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830185.364914:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830185.364915:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830185.365016:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518219520:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830185.365072:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830185.365115:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830185/real 1787830185] req@ffff8b2e88076d00 x1874672518219520/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830185.365132:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518219520/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830185.365141:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830185.365145:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830185.365148:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518219520:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830185.365198:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3191 00000100:00080000:1.0:1787830185.365203:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830185.365207:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830185.365216:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830185.365230:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518219648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830185.365247:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830185.365250:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830185.365252:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830185.365256:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3191 00000100:00080000:1.0:1787830185.365259:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830185.365262:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830185.365267:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830185.365277:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518219776/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830185.365286:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830185.365289:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830185.365290:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830185.365294:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3191 00000100:00080000:1.0:1787830185.365297:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830185.365298:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830185.365300:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830185.365303:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830185.365313:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518219904/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830185.365321:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830185.365324:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830185.365325:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830185.365329:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3191 00000100:00080000:1.0:1787830185.365332:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830185.365335:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830185.365339:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830185.365348:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518220032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830185.365374:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830185.365378:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830185.365380:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830185.365383:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3191 00000100:00080000:1.0:1787830185.365387:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830185.365392:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830185.365396:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830185.365401:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830185.365408:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518220160/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830185.365417:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830185.365420:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830185.365422:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830185.365425:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3191 00000100:00080000:1.0:1787830185.365428:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830185.365432:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830185.365435:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830185.365444:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518220288/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830185.365518:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518219648:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830185.365542:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830185.365554:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830185/real 1787830185] req@ffff8b2e88077c00 x1874672518219648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830185.365565:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518219648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830185.365573:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830185.365576:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830185.365579:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518219648:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830185.365584:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518219776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830185.365604:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830185.365614:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830185/real 1787830185] req@ffff8b2e88075680 x1874672518219776/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830185.365625:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518219776/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830185.365632:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830185.365634:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830185.365637:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518219776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830185.365643:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518219904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830185.365682:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830185.365693:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830185/real 1787830185] req@ffff8b2e88074780 x1874672518219904/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830185.365703:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518219904/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830185.365710:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830185.365713:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830185.365716:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518219904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830185.365721:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518220032:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830185.365738:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830185.365748:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830185/real 1787830185] req@ffff8b2e88074f00 x1874672518220032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830185.365766:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518220032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830185.365774:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830185.365776:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830185.365779:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518220032:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830185.365784:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518220160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830185.365801:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830185.365812:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830185/real 1787830185] req@ffff8b2e88075a40 x1874672518220160/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830185.365835:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518220160/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830185.365842:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830185.365845:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830185.365847:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518220160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830185.365852:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518220288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830185.365869:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830185.365880:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830185/real 1787830185] req@ffff8b2e88077480 x1874672518220288/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830185.365890:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518220288/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830210 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830185.365902:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830185.365906:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830185.365908:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518220288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830185.685239:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830186.197640:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830186.201102:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830186.388781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830186.388781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830186.708844:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830187.220769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830187.222644:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830187.412786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830187.413120:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830187.732791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830188.245045:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830188.245254:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830188.436950:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830188.437176:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830188.756869:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830189.269216:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830189.270927:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830189.460801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830189.460821:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830189.780754:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830190.292766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830190.293366:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830190.484776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830190.484815:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830190.485185:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830190.485190:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830190.485194:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830190.485203:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3196 00000100:00080000:3.0:1787830190.485215:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830190.485222:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830190.485336:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518220416/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830190.485365:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830190.485369:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830190.485371:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830190.485375:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3196 00000100:00080000:3.0:1787830190.485379:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830190.485409:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518220416:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830190.485521:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830190.485533:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830190.488506:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518220544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830190.488554:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830190.488559:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830190.488562:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830190.488569:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3196 00000100:00080000:3.0:1787830190.488579:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830190.488586:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830190.488599:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830190.488605:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518220544:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830190.488633:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518220672/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830190.488644:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830190.488668:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830190.488670:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830190.488673:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3196 00000100:00080000:3.0:1787830190.488813:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830190.488820:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830190.488827:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830190.488834:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518220672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830190.488843:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518220800/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830190.488857:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830190.488861:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830190.488863:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830190.488868:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3196 00000100:00100000:2.0:1787830190.488871:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518220800:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830190.488873:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830190.488881:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830190.488887:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830190.488901:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518220928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830190.488915:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830190.488919:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830190.488921:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830190.488925:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3196 00000100:00100000:2.0:1787830190.488930:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518220928:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830190.488932:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830190.488941:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830190.488946:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830190.488959:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518221056/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830190.488971:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830190.488974:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830190.488976:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830190.488981:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3196 00000100:00080000:3.0:1787830190.488985:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830190.488989:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518221056:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830190.488993:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830190.489000:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830190.489016:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518221184/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830190.489109:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518221184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830190.805026:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830191.316900:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830191.317957:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830191.508806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830191.509149:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830191.829052:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830192.340804:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830192.340945:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830192.532800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830192.532948:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830192.852797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830193.364810:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830193.364909:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830193.556774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830193.556776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:1.0:1787830193.557289:0:2336:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787830193.557329:0:2336:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787830193.557344:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830193.557382:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830193.557393:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830193.557401:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830193.557410:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830193.557420:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:2.0:1787830193.557429:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830190/real 1787830193] req@ffff8b2e8803a1c0 x1874672518220928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830193.557452:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518220928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787830193.557460:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00080000:2.0:1787830193.557467:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830193.557472:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830193.557476:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518220928:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830193.557483:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830190/real 1787830193] req@ffff8b2e8803b480 x1874672518220800/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830193.557495:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518220800/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830193.557502:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830193.557505:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830193.557508:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518220800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830193.557514:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830190/real 1787830193] req@ffff8b2e880392c0 x1874672518220672/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830193.557525:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518220672/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830193.557533:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830193.557538:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830193.557541:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518220672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830193.557547:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830190/real 1787830193] req@ffff8b2e88039680 x1874672518220544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830193.557557:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518220544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830193.557564:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830193.557567:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830193.557570:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518220544:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830193.557576:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830190/real 1787830193] req@ffff8b2e880383c0 x1874672518220416/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830193.557588:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518220416/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830193.557594:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830193.557597:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830193.557600:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518220416:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830193.557627:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830190/real 1787830193] req@ffff8b2e8803b840 x1874672518221056/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830193.557640:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518221056/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830193.557668:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830193.557671:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830193.557674:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518221056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830193.557684:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830190/real 1787830193] req@ffff8b2e8803a580 x1874672518221184/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830193.557695:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518221184/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830215 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830193.557703:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830193.557706:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830193.557709:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518221184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830193.876836:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830194.390252:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830194.390455:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830194.580815:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830194.581004:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830194.900978:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830195.412804:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830195.414896:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830195.604952:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787830195.605114:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830195.605120:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830195.605123:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830195.605132:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3202 00000100:00080000:2.0:1787830195.605141:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830195.605147:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830195.605181:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518221312/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830195.605252:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518221312:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830195.605318:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787830195.605777:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830195.605781:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830195.605783:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830195.605788:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3202 00000100:00080000:2.0:1787830195.605794:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830195.605799:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830195.605808:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830195.605823:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518221440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830195.605892:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830195.609689:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830195/real 1787830195] req@ffff8b2e8801de00 x1874672518221312/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830195.609718:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518221312/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830195.609729:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830195.609733:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830195.609738:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518221312:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830195.609777:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518221440:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830195.609833:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830195.609850:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830195/real 1787830195] req@ffff8b2e8801cb40 x1874672518221440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830195.609862:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518221440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830195.609870:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830195.609874:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830195.609877:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518221440:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787830195.610583:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830195.610591:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830195.610593:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830195.610600:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3202 00000100:00080000:2.0:1787830195.610607:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830195.610613:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830195.610624:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830195.610662:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518221568/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830195.610676:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830195.610679:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830195.610681:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830195.610683:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3202 00000100:00080000:2.0:1787830195.610687:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830195.610690:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830195.610695:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830195.610704:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518221696/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830195.610713:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830195.610717:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830195.610718:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830195.610723:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3202 00000100:00080000:2.0:1787830195.610726:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830195.610729:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830195.610733:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830195.610742:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518221824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830195.610760:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830195.610763:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830195.610764:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830195.610768:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3202 00000100:00080000:2.0:1787830195.610771:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830195.610774:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830195.610777:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830195.610787:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518221952/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830195.610794:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830195.610796:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830195.610798:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830195.610800:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3202 00000100:00080000:2.0:1787830195.610803:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830195.610806:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830195.610809:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830195.610817:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518222080/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830195.610948:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518221568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830195.610984:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830195.611000:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830195/real 1787830195] req@ffff8b2e8801c3c0 x1874672518221568/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830195.612155:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518221568/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830195.612171:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830195.612175:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830195.612179:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518221568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830195.612189:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518221696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830195.612230:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830195.612248:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830195/real 1787830195] req@ffff8b2e8801d680 x1874672518221696/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830195.612260:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518221696/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830195.612267:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830195.612270:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830195.612273:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518221696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830195.612279:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518221824:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830195.612298:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830195.612308:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830195/real 1787830195] req@ffff8b2e8801f0c0 x1874672518221824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830195.612319:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518221824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830195.612333:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830195.612336:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830195.612339:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518221824:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830195.612345:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518221952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830195.612365:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830195.612375:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830195/real 1787830195] req@ffff8b2e8801c780 x1874672518221952/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830195.612385:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518221952/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830195.612392:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830195.612395:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830195.612398:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518221952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830195.612404:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518222080:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830195.612420:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830195.612430:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830195/real 1787830195] req@ffff8b2e8801ed00 x1874672518222080/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830195.612439:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518222080/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830220 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830195.612446:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830195.612448:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830195.612452:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518222080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830195.925281:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830196.436844:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830196.436997:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830196.629357:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830196.630117:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830196.949480:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830197.460808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830197.460922:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830197.652760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830197.652760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830197.972894:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830198.484868:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830198.484906:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830198.676822:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830198.677117:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830198.996779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830199.508885:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830199.511526:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830199.700784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830199.701067:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830200.020776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830200.532807:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830200.533526:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830200.724785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830200.725250:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830200.725363:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830200.725368:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830200.725372:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830200.725381:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3207 00000100:00080000:0.0:1787830200.725391:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830200.725397:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830200.725447:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518222208/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830200.725480:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830200.725483:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830200.725485:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830200.725489:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3207 00000100:00080000:0.0:1787830200.725492:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830200.725496:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830200.725505:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830200.725518:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518222336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830200.725527:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830200.725530:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830200.725532:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830200.725535:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3207 00000100:00080000:0.0:1787830200.725538:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830200.725541:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830200.725545:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830200.725553:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518222464/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830200.725563:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830200.725566:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830200.725567:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830200.725570:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3207 00000100:00080000:0.0:1787830200.725572:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830200.725575:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830200.725578:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830200.725586:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518222592/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830200.725594:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830200.725597:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830200.725599:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830200.725602:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3207 00000100:00080000:0.0:1787830200.725605:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830200.725608:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830200.725612:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830200.725632:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518222720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830200.725641:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830200.725644:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830200.725662:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830200.725666:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3207 00000100:00080000:0.0:1787830200.725669:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830200.725672:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830200.725676:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830200.725686:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518222848/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830200.725694:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830200.725698:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830200.725699:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830200.725702:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3207 00000100:00080000:0.0:1787830200.725705:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830200.725707:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830200.725722:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830200.725732:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518222976/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830200.726083:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518222208:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830200.726137:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830200.726179:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830200/real 1787830200] req@ffff8b2e880403c0 x1874672518222208/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830200.726194:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518222208/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830200.726203:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830200.726206:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830200.726210:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518222208:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830200.726217:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518222336:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830200.726238:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830200.726250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830200/real 1787830200] req@ffff8b2e88040000 x1874672518222336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830200.726261:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518222336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830200.726269:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830200.726272:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830200.726275:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518222336:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830200.726281:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518222464:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830200.726300:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830200.726310:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830200/real 1787830200] req@ffff8b2e880430c0 x1874672518222464/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830200.726321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518222464/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830200.726328:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830200.726331:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830200.726334:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518222464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830200.726340:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518222592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830200.726358:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830200.726369:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830200/real 1787830200] req@ffff8b2e88043480 x1874672518222592/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830200.726380:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518222592/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830200.726387:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830200.726390:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830200.726392:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518222592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830200.726398:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518222720:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830200.726415:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830200.726426:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830200/real 1787830200] req@ffff8b2e88042940 x1874672518222720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830200.726437:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518222720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830200.726444:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830200.726447:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830200.726449:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518222720:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830200.726455:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518222848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830200.726469:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830200.726477:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830200/real 1787830200] req@ffff8b2e88040f00 x1874672518222848/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830200.726488:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518222848/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830200.726495:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830200.726498:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830200.726501:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518222848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830200.726506:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518222976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830200.726523:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830200.726533:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830200/real 1787830200] req@ffff8b2e88041a40 x1874672518222976/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830200.726542:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518222976/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830225 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830200.726550:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830200.726564:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830200.726566:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518222976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830201.045497:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830201.556783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830201.556885:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830201.748966:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830201.749542:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830202.069059:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830202.581074:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830202.581264:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830202.772792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830202.772792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830203.092781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830203.604795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830203.605367:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830203.796786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830203.796786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830204.116744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830204.629032:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830204.630513:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830204.820738:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830204.820745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830205.140801:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830205.652834:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830205.653214:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830205.844827:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830205.844827:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830205.844933:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830205.844940:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830205.844944:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830205.844954:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3212 00000100:00080000:1.0:1787830205.844965:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830205.844973:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830205.845012:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518223104/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830205.845045:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830205.845048:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830205.845050:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830205.845054:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3212 00000100:00080000:1.0:1787830205.845058:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830205.845061:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830205.845071:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830205.845085:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518223232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830205.845095:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830205.845098:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830205.845100:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830205.845103:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3212 00000100:00080000:1.0:1787830205.845106:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830205.845110:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830205.845114:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830205.845125:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518223360/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830205.845133:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830205.845136:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830205.845137:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830205.845140:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3212 00000100:00080000:1.0:1787830205.845144:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830205.845147:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830205.845151:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830205.845160:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518223488/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830205.845168:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830205.845171:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830205.845173:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830205.845176:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3212 00000100:00080000:1.0:1787830205.845180:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830205.845183:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830205.845186:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830205.845195:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518223616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830205.845203:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830205.845205:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830205.845207:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830205.845210:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3212 00000100:00080000:1.0:1787830205.845213:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830205.845216:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830205.845221:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830205.845229:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518223744/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830205.845236:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830205.845239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830205.845241:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830205.845243:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3212 00000100:00080000:1.0:1787830205.845246:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830205.845249:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830205.845253:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830205.845262:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518223872/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830205.845469:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518223104:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830205.845524:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830205.845563:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830205/real 1787830205] req@ffff8b2e88076940 x1874672518223104/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830205.845578:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518223104/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830205.845587:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830205.845591:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830205.845595:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518223104:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830205.845601:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518223232:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830205.845621:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830205.845633:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830205/real 1787830205] req@ffff8b2e88077480 x1874672518223232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830205.845644:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518223232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830205.845671:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830205.845674:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830205.845687:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518223232:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830205.845693:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518223360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830205.845713:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830205.845722:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830205/real 1787830205] req@ffff8b2e88075a40 x1874672518223360/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830205.845732:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518223360/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830205.845739:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830205.845742:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830205.845745:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518223360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830205.845751:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518223488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830205.845766:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830205.845774:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830205/real 1787830205] req@ffff8b2e88074f00 x1874672518223488/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830205.845784:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518223488/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830205.845790:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830205.845793:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830205.845796:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518223488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830205.845800:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518223616:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830205.845818:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830205.845828:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830205/real 1787830205] req@ffff8b2e88074780 x1874672518223616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830205.845838:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518223616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830205.845846:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830205.845849:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830205.845852:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518223616:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830205.845857:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518223744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830205.845875:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830205.845887:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830205/real 1787830205] req@ffff8b2e88075680 x1874672518223744/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830205.845898:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518223744/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830205.845911:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830205.845915:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830205.845918:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518223744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830205.845923:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518223872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830205.845941:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830205.845951:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830205/real 1787830205] req@ffff8b2e88077c00 x1874672518223872/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830205.845960:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518223872/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830230 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830205.845968:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830205.845970:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830205.845973:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518223872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830206.166781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830206.676880:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830206.677086:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830206.868789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830206.869324:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830207.188900:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830207.700775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830207.701560:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830207.892790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830207.892841:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830208.212830:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830208.724840:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830208.724965:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830208.916797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830208.917373:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830209.236817:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830209.748959:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830209.749724:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830209.940794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830209.940794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830210.260871:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830210.772849:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830210.772886:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830210.964788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830210.964788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830210.964788:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830210.964800:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830210.964804:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830210.964815:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3217 00000100:00080000:3.0:1787830210.964826:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830210.964833:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830210.964870:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518224000/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830210.964892:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830210.964895:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830210.964896:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830210.964899:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3217 00000100:00080000:3.0:1787830210.964901:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830210.964903:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830210.964910:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830210.964918:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518224128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830210.964923:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830210.964925:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830210.964926:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830210.964928:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3217 00000100:00080000:3.0:1787830210.964930:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830210.964932:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830210.964934:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830210.964938:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518224256/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830210.964943:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830210.964945:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830210.964946:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830210.964948:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3217 00000100:00080000:3.0:1787830210.964950:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830210.964952:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830210.964955:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830210.964959:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518224384/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830210.964964:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830210.964965:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830210.964966:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830210.964968:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3217 00000100:00080000:3.0:1787830210.964969:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830210.964971:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830210.964973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830210.964979:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518224512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830210.964983:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830210.964985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830210.964986:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830210.964988:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3217 00000100:00080000:3.0:1787830210.964990:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830210.964991:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830210.964994:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830210.964999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518224640/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830210.965004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830210.965006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830210.965007:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830210.965009:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3217 00000100:00080000:3.0:1787830210.965010:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830210.965012:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830210.965014:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830210.965018:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518224768/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830210.965165:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518224000:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830210.965221:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830210.965261:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830210/real 1787830210] req@ffff8b2e88038f00 x1874672518224000/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830210.965288:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518224000/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830210.965298:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830210.965302:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830210.965306:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518224000:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830210.965314:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518224128:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830210.965337:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830210.965350:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830210/real 1787830210] req@ffff8b2e8803b0c0 x1874672518224128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830210.965360:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518224128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830210.965369:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830210.965372:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830210.965375:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518224128:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830210.965380:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518224256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830210.965400:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830210.965411:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830210/real 1787830210] req@ffff8b2e88039a40 x1874672518224256/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830210.965423:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518224256/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830210.965435:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830210.965438:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830210.965442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518224256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830210.965448:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518224384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830210.965466:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830210.965476:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830210/real 1787830210] req@ffff8b2e88039e00 x1874672518224384/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830210.965485:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518224384/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830210.965493:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830210.965496:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830210.965499:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518224384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830210.965504:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518224512:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830210.965523:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830210.965533:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830210/real 1787830210] req@ffff8b2e8803a580 x1874672518224512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830210.965544:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518224512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830210.965552:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830210.965555:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830210.965558:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518224512:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830210.965564:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518224640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830210.965583:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830210.965592:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830210/real 1787830210] req@ffff8b2e8803b840 x1874672518224640/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830210.965603:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518224640/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830210.965610:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830210.965612:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830210.965615:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518224640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830210.965620:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518224768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830210.965637:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830210.965680:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830210/real 1787830210] req@ffff8b2e880383c0 x1874672518224768/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830210.965691:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518224768/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830235 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830210.965697:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830210.965700:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830210.965703:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518224768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830211.285160:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830211.796787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830211.796850:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830211.988780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830211.989706:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830212.309891:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830212.820815:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830212.820947:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830213.012868:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830213.013546:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830213.332837:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830213.844821:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830213.844968:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830214.037083:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830214.037190:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830214.356848:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830214.868780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830214.868815:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830215.060916:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830215.061686:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830215.380763:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830215.892778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830215.892997:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830216.084772:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830216.084775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830216.084785:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830216.084790:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830216.084801:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3222 00000100:00100000:2.0:1787830216.084807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830216.084815:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830216.084824:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830216.084865:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518224896/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830216.084893:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830216.084897:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830216.084899:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830216.084904:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3222 00000100:00080000:3.0:1787830216.084909:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830216.084918:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830216.084929:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518224896:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787830216.084930:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830216.084961:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518225024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830216.084976:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830216.084979:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830216.084981:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830216.084985:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3222 00000100:00080000:3.0:1787830216.084996:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830216.085003:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830216.085009:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1787830216.085012:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830216.085025:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518225152/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830216.085035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830216.085038:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830216.085040:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830216.085044:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3222 00000100:00080000:3.0:1787830216.085048:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830216.085053:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830216.085058:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830216.085063:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830216/real 1787830216] req@ffff8b2e88039680 x1874672518224896/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830216.085073:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518225280/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830216.085084:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518224896/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830216.085100:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830216.085104:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787830216.085106:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830216.085110:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518224896:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830216.085111:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830216.085116:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830216.085122:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3222 00000100:00080000:3.0:1787830216.085127:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830216.085130:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830216.085137:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518225024:192.168.203.101@tcp:38:kworker.0 00000008:00100000:3.0:1787830216.085143:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787830216.085165:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830216.085177:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518225408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830216.085183:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830216/real 1787830216] req@ffff8b2e880392c0 x1874672518225024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830216.085191:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830216.085198:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830216.085200:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830216.085205:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518225024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830216.085208:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3222 00000100:00080000:3.0:1787830216.085215:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830216.085222:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787830216.085223:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787830216.085226:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830216.085245:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518225536/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830216.085246:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830216.085257:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518225024:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830216.085264:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830216.085267:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518225152:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830216.085269:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830216.085272:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830216.085278:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3222 00000100:00080000:3.0:1787830216.085282:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830216.085286:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830216.085290:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787830216.085301:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830216.085309:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518225664/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830216.085317:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830216/real 1787830216] req@ffff8b2e8803b480 x1874672518225152/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830216.085332:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518225152/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830216.085348:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830216.085352:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830216.085355:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518225152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830216.085361:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518225280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830216.085381:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830216.085392:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830216/real 1787830216] req@ffff8b2e8803a1c0 x1874672518225280/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830216.085409:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518225280/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830216.085418:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830216.085424:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830216.085427:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518225280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830216.085442:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518225408:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830216.085461:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830216.085472:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830216/real 1787830216] req@ffff8b2e88039680 x1874672518225408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830216.085482:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518225408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830216.085491:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830216.085494:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830216.085497:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518225408:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830216.085502:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518225536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830216.085518:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830216.085527:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830216/real 1787830216] req@ffff8b2e880383c0 x1874672518225536/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830216.085539:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518225536/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830216.085546:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830216.085548:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830216.085551:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518225536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830216.085556:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518225664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830216.085574:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830216.085596:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830216/real 1787830216] req@ffff8b2e8803b840 x1874672518225664/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830216.085621:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518225664/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830241 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830216.085630:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830216.085633:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830216.085636:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518225664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830216.404811:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830216.916800:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830216.916825:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830217.108810:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830217.109124:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830217.429552:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830217.940774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830217.941171:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830218.132771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830218.132843:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830218.452906:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830218.964779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830218.964973:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830219.156846:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830219.157058:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830219.476792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830219.988871:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830219.989716:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830220.180823:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830220.180936:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830220.500792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830221.012776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830221.012817:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830221.204917:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830221.204922:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830221.204969:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830221.204979:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830221.204983:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830221.204993:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3227 00000100:00080000:1.0:1787830221.205012:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830221.205020:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830221.205059:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518225792/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830221.205122:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518225792:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830221.205187:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830221.205924:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830221.205929:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830221.205932:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830221.205938:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3227 00000100:00080000:1.0:1787830221.205943:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830221.205949:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830221.205960:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830221.205978:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518225920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830221.209322:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830221/real 1787830221] req@ffff8b2e88076d00 x1874672518225792/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830221.209350:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518225792/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830221.209362:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830221.209367:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830221.209371:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518225792:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830221.209401:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518225920:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830221.209452:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830221.209468:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830221/real 1787830221] req@ffff8b2e88077840 x1874672518225920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830221.209480:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518225920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830221.209488:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830221.209491:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830221.209494:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518225920:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830221.210242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830221.210249:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830221.210252:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830221.210258:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3227 00000100:00080000:1.0:1787830221.210264:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830221.210270:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830221.210280:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830221.210305:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518226048/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830221.210318:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830221.210321:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830221.210323:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830221.210326:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3227 00000100:00080000:1.0:1787830221.210329:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830221.210332:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830221.210338:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830221.210347:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518226176/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830221.210355:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830221.210358:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830221.210360:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830221.210363:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3227 00000100:00080000:1.0:1787830221.210366:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830221.210370:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830221.210374:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830221.210384:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518226304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830221.210392:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830221.210395:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830221.210396:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830221.210400:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3227 00000100:00080000:1.0:1787830221.210403:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830221.210406:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830221.210410:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830221.210419:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518226432/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830221.210427:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830221.210430:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830221.210432:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830221.210435:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3227 00000100:00080000:1.0:1787830221.210438:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830221.210441:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830221.210446:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830221.210453:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518226560/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830221.210800:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518226048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830221.210940:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830221.210959:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830221/real 1787830221] req@ffff8b2e880770c0 x1874672518226048/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830221.210972:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518226048/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830221.210981:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830221.210984:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830221.210987:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518226048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830221.210993:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518226176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830221.211689:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830221.211704:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830221/real 1787830221] req@ffff8b2e880761c0 x1874672518226176/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830221.211715:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518226176/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830221.211784:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830221.211788:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830221.211791:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518226176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830221.211798:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518226304:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830221.211819:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830221.211985:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830221/real 1787830221] req@ffff8b2e88074000 x1874672518226304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830221.212056:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518226304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830221.212067:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830221.212071:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830221.212074:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518226304:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830221.212081:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518226432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830221.212104:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830221.212116:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830221/real 1787830221] req@ffff8b2e88074b40 x1874672518226432/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830221.212128:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518226432/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830221.212136:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830221.212139:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830221.212231:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518226432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830221.212240:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518226560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830221.212262:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830221.212273:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830221/real 1787830221] req@ffff8b2e88075e00 x1874672518226560/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830221.212285:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518226560/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830246 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830221.212293:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830221.212296:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830221.212299:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518226560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830221.525363:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830222.037403:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830222.037810:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830222.228779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830222.229403:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830222.549318:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830223.060924:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830223.063676:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830223.252990:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830223.254366:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830223.572880:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830224.085063:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830224.085595:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830224.276781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830224.276781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830224.596994:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830225.109041:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830225.112740:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830225.301754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830225.302906:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830225.620805:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830226.132869:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830226.132956:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:0.0:1787830226.324774:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830226.324776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830226.324785:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830226.324789:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830226.324800:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3232 00000100:00080000:0.0:1787830226.324811:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830226.324818:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830226.324855:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518226688/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830226.324877:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830226.324881:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830226.324882:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830226.324886:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3232 00000100:00080000:0.0:1787830226.324889:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830226.324893:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830226.324904:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830226.324917:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518226816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830226.324938:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830226.324942:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830226.324944:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830226.324947:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3232 00000100:00080000:0.0:1787830226.324952:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830226.324952:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518226688:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787830226.324960:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830226.324965:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830226.324977:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830226.324978:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518226944/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830226.324990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830226.324993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830226.324996:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830226.324999:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3232 00000100:00080000:0.0:1787830226.325006:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830226.325009:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830226.325013:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787830226.325017:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830226.325026:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518227072/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830226.325036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830226.325039:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830226.325040:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830226.325054:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3232 00000100:00080000:0.0:1787830226.325058:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830226.325062:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830226.325066:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830226.325077:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518227200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830226.325085:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830226.325088:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830226.325089:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830226.325092:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3232 00000100:00080000:0.0:1787830226.325096:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830226.325098:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830226.325102:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830226.325110:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518227328/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830226.325117:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830226.325120:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830226.325121:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830226.325124:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3232 00000100:00080000:0.0:1787830226.325127:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830226.325130:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830226.325133:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830226.325140:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518227456/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830226.325245:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830226/real 1787830226] req@ffff8b2e880421c0 x1874672518226688/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830226.325264:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518226688/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830226.325274:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830226.325279:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830226.325282:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518226688:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830226.325289:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518226816:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830226.325315:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830226.325328:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830226/real 1787830226] req@ffff8b2e88040b40 x1874672518226816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830226.325339:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518226816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830226.325346:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830226.325349:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830226.325352:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518226816:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830226.325372:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518226944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830226.325390:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830226.325402:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830226/real 1787830226] req@ffff8b2e880412c0 x1874672518226944/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830226.325418:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518226944/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830226.325426:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830226.325428:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830226.325431:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518226944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830226.325437:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518227072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830226.325454:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830226.325463:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830226/real 1787830226] req@ffff8b2e88043840 x1874672518227072/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830226.325473:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518227072/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830226.325479:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830226.325482:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830226.325485:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518227072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830226.325490:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518227200:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830226.325506:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830226.325515:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830226/real 1787830226] req@ffff8b2e88040780 x1874672518227200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830226.325526:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518227200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830226.325548:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830226.325552:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830226.325555:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518227200:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830226.325561:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518227328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830226.325578:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830226.325588:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830226/real 1787830226] req@ffff8b2e88042d00 x1874672518227328/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830226.325606:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518227328/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830226.325613:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830226.325616:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830226.325619:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518227328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830226.325625:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518227456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830226.325643:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830226.325674:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830226/real 1787830226] req@ffff8b2e88042580 x1874672518227456/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830226.325685:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518227456/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830251 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830226.325692:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830226.325696:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830226.325699:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518227456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830226.644790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830227.157069:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830227.162120:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830227.348782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830227.348795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830227.668803:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830228.180756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830228.180868:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830228.373037:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830228.373045:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830228.692802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830229.204903:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830229.204960:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830229.397185:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830229.397693:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830229.716882:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830230.228832:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830230.228947:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830230.420824:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830230.422675:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830230.740928:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830231.252754:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830231.253082:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830231.444810:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830231.444822:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830231.444826:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830231.444836:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3237 00000100:00080000:0.0:1787830231.444846:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830231.444861:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830231.444903:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518227584/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830231.444930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830231.444934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830231.444936:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830231.444940:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3237 00000100:00080000:0.0:1787830231.444944:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830231.444947:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830231.444957:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830231.444973:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518227712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830231.444982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830231.444985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830231.444988:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830231.444992:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3237 00000100:00100000:1.0:1787830231.444995:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830231.444996:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830231.445002:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830231.445008:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830231.445019:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518227840/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830231.445028:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830231.445031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830231.445033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830231.445037:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3237 00000100:00080000:0.0:1787830231.445040:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830231.445043:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830231.445047:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830231.445057:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518227968/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830231.445064:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830231.445067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830231.445069:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830231.445072:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3237 00000100:00080000:0.0:1787830231.445075:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830231.445078:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830231.445082:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830231.445091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518228096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830231.445098:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830231.445101:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830231.445103:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830231.445106:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3237 00000100:00080000:0.0:1787830231.445108:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830231.445111:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830231.445115:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830231.445124:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518228224/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830231.445132:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830231.445134:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830231.445136:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830231.445139:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3237 00000100:00080000:0.0:1787830231.445158:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830231.445161:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830231.445165:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830231.445174:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518228352/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830231.447310:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518227584:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830231.447375:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830231.447421:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830231/real 1787830231] req@ffff8b2e88041e00 x1874672518227584/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830231.447443:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518227584/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830231.447452:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830231.447456:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830231.447460:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518227584:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830231.447467:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518227712:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830231.447489:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830231.447513:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830231/real 1787830231] req@ffff8b2e88042580 x1874672518227712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830231.447524:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518227712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830231.447532:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830231.447535:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830231.447538:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518227712:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830231.447544:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518227840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830231.447570:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830231.447581:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830231/real 1787830231] req@ffff8b2e88042d00 x1874672518227840/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830231.447592:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518227840/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830231.447606:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830231.447609:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830231.447612:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518227840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830231.447618:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518227968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830231.447770:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830231.447783:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830231/real 1787830231] req@ffff8b2e88040780 x1874672518227968/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830231.447796:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518227968/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830231.447803:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830231.447807:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830231.447810:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518227968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830231.447816:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518228096:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830231.447836:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830231.447847:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830231/real 1787830231] req@ffff8b2e88043840 x1874672518228096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830231.447859:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518228096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830231.447867:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830231.447870:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830231.447873:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518228096:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830231.447878:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518228224:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830231.447898:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830231.447909:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830231/real 1787830231] req@ffff8b2e880412c0 x1874672518228224/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830231.447921:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518228224/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830231.447928:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830231.447931:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830231.447935:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518228224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830231.447942:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518228352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830231.447961:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830231.447972:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830231/real 1787830231] req@ffff8b2e88040b40 x1874672518228352/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830231.447983:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518228352/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830256 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830231.447990:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830231.447993:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830231.447996:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518228352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830231.450182:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830231.764791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830232.276998:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830232.277149:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830232.468783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830232.468863:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830232.789812:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830233.300790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830233.300863:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830233.492825:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830233.492935:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830233.812783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830234.325029:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830234.325448:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830234.516798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830234.516842:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830234.836853:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830235.348785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830235.350770:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830235.540826:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830235.542305:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830235.860843:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830236.372779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830236.372916:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830236.564800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830236.565332:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830236.565342:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830236.565346:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830236.565355:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3242 00000100:00080000:3.0:1787830236.565365:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830236.565373:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830236.565413:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518228480/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830236.565448:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830236.565452:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830236.565453:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830236.565458:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3242 00000100:00080000:3.0:1787830236.565461:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830236.565474:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830236.565484:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830236.565495:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518228608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830236.565504:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830236.565508:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830236.565509:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830236.565512:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3242 00000100:00080000:3.0:1787830236.565516:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830236.565519:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830236.565523:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830236.565534:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518228736/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830236.565542:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830236.565546:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830236.565547:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830236.565551:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3242 00000100:00080000:3.0:1787830236.565554:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830236.565558:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830236.565562:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830236.565570:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518228480:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830236.565575:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518228864/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830236.565587:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830236.565591:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830236.565592:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830236.565596:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3242 00000100:00080000:3.0:1787830236.565600:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830236.565603:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830236.565608:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830236.565618:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518228992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830236.565626:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830236.565629:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830236.565631:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787830236.565636:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830236.565637:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3242 00000100:00080000:3.0:1787830236.565643:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830236.565674:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830236.565679:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830236.565691:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518229120/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830236.565706:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830236.565710:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830236.565712:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830236.565714:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830236/real 1787830236] req@ffff8b2e8803a580 x1874672518228480/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830236.565719:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3242 00000100:00080000:3.0:1787830236.565725:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830236.565732:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830236.565737:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830236.565741:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518228480/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830236.565751:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518229248/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830236.565753:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830236.565761:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830236.565768:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518228480:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830236.565778:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518228608:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830236.565802:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830236.565814:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830236/real 1787830236] req@ffff8b2e88039e00 x1874672518228608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830236.565903:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518228608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830236.565913:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830236.565916:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830236.565919:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518228608:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830236.565925:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518228736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830236.565953:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830236.565965:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830236/real 1787830236] req@ffff8b2e88039a40 x1874672518228736/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830236.565976:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518228736/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830236.565984:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830236.565987:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830236.565990:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518228736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830236.566014:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518228864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830236.566033:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830236.566055:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830236/real 1787830236] req@ffff8b2e8803b0c0 x1874672518228864/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830236.566067:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518228864/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830236.566075:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830236.566077:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830236.566080:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518228864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830236.566086:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518228992:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830236.566104:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830236.566115:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830236/real 1787830236] req@ffff8b2e88038f00 x1874672518228992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830236.566132:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518228992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830236.566141:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830236.566144:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830236.566147:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518228992:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830236.566153:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518229120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830236.566170:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830236.566180:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830236/real 1787830236] req@ffff8b2e8803b840 x1874672518229120/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830236.566191:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518229120/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830236.566199:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830236.566202:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830236.566205:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518229120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830236.566211:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518229248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830236.566227:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830236.566236:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830236/real 1787830236] req@ffff8b2e880383c0 x1874672518229248/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830236.566247:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518229248/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830261 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830236.566259:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830236.566261:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830236.566264:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518229248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830236.566569:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830236.884807:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830237.396753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830237.396904:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830237.588799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830237.588901:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830237.908808:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830238.420797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830238.420836:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830238.612781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830238.612845:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830238.932781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830239.444788:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830239.444834:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830239.636784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830239.640133:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830239.956784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830240.468810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830240.470392:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830240.660748:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830240.660748:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830240.982559:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830241.493512:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830241.496574:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:3.0:1787830241.684739:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830241.684743:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830241.684743:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830241.684753:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830241.684759:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830241.684780:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3248 00000100:00080000:3.0:1787830241.684792:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830241.684800:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830241.684846:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518229376/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830241.684869:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830241.684872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830241.684874:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830241.684877:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3248 00000100:00080000:3.0:1787830241.684881:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830241.684886:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830241.684895:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830241.684909:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518229376:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830241.684910:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518229504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830241.684924:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830241.684926:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830241.684932:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830241.684936:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3248 00000100:00080000:3.0:1787830241.684939:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830241.684942:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830241.684947:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830241.684957:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518229632/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830241.684966:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830241.684969:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830241.684971:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830241.684974:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3248 00000100:00080000:3.0:1787830241.684977:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830241.684980:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:2.0:1787830241.684986:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:3.0:1787830241.684987:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830241.684999:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518229760/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830241.685013:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830241.685016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830241.685017:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830241.685021:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3248 00000100:00080000:3.0:1787830241.685025:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830241.685029:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830241.685033:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830241.685037:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830241/real 1787830241] req@ffff8b2e88039680 x1874672518229376/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830241.685047:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518229888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830241.685057:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830241.685060:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518229376/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830241.685071:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830241.685074:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830241.685074:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830241.685082:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3248 00000100:00080000:2.0:1787830241.685085:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787830241.685087:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830241.685093:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518229376:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830241.685094:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830241.685100:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830241.685111:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518230016/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830241.685122:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830241.685126:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518229504:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830241.685127:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830241.685130:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830241.685137:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3248 00000100:00080000:3.0:1787830241.685141:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830241.685144:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830241.685149:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830241.685159:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518230144/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830241.685162:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830241.685183:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830241/real 1787830241] req@ffff8b2e8803a1c0 x1874672518229504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830241.685196:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518229504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830241.685205:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830241.685208:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830241.685212:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518229504:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830241.685218:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518229632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830241.685238:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830241.685250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830241/real 1787830241] req@ffff8b2e8803b480 x1874672518229632/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830241.685261:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518229632/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830241.685269:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830241.685273:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830241.685275:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518229632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830241.685281:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518229760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830241.685300:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830241.685310:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830241/real 1787830241] req@ffff8b2e880392c0 x1874672518229760/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830241.685320:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518229760/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830241.685327:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830241.685330:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830241.685333:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518229760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830241.685339:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518229888:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830241.685357:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830241.685367:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830241/real 1787830241] req@ffff8b2e880383c0 x1874672518229888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830241.685378:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518229888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830241.685386:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830241.685390:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830241.685393:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518229888:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830241.685409:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518230016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830241.685425:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830241.685448:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830241/real 1787830241] req@ffff8b2e8803b840 x1874672518230016/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830241.685459:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518230016/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830241.685467:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830241.685470:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830241.685473:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518230016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830241.685479:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518230144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830241.685497:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830241.685507:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830241/real 1787830241] req@ffff8b2e88038f00 x1874672518230144/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830241.685518:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518230144/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830266 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830241.685525:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830241.685528:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830241.685531:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518230144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830242.004950:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830242.516769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830242.516818:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830242.708719:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830242.709620:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830243.028796:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830243.540790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830243.544081:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830243.732858:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830243.733481:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830244.052798:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830244.564788:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830244.565232:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830244.757066:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830244.757337:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830245.076777:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830245.588804:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830245.588853:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830245.780790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830245.780823:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830246.100760:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830246.614094:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830246.614479:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:3.0:1787830246.804779:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830246.804782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830246.804791:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830246.804797:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830246.804807:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3253 00000100:00080000:3.0:1787830246.804818:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830246.804826:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830246.804876:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518230272/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830246.804909:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830246.804912:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830246.804914:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830246.804918:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3253 00000100:00080000:3.0:1787830246.804921:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830246.804923:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830246.804933:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830246.804944:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518230400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830246.804958:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830246.804961:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830246.804963:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830246.804966:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3253 00000100:00080000:3.0:1787830246.804969:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830246.804972:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830246.804977:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830246.805002:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518230528/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830246.805016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830246.805019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830246.805021:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830246.805023:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3253 00000100:00080000:3.0:1787830246.805030:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830246.805033:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830246.805038:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830246.805048:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518230656/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830246.805057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830246.805060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830246.805062:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830246.805066:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3253 00000100:00100000:2.0:1787830246.805072:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518230272:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830246.805073:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830246.805081:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830246.805087:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830246.805100:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518230784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830246.805110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830246.805113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830246.805115:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830246.805118:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3253 00000100:00080000:3.0:1787830246.805122:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830246.805126:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830246.805130:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787830246.805138:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830246.805142:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518230912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830246.805151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830246.805155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830246.805156:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830246.805161:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3253 00000100:00080000:3.0:1787830246.805165:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830246.805168:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830246.805173:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830246.805183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518231040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830246.805195:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830246/real 1787830246] req@ffff8b2e8803b0c0 x1874672518230272/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830246.805223:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518230272/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830246.805232:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830246.805235:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830246.805238:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518230272:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830246.805245:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518230400:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830246.805264:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830246.805273:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830246/real 1787830246] req@ffff8b2e88039a40 x1874672518230400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830246.805283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518230400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830246.805291:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830246.805293:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830246.805296:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518230400:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830246.805300:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518230528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830246.805315:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830246.805324:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830246/real 1787830246] req@ffff8b2e88039e00 x1874672518230528/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830246.805335:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518230528/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830246.805348:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830246.805351:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830246.805354:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518230528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830246.805360:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518230656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830246.805383:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830246.805402:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830246/real 1787830246] req@ffff8b2e8803a580 x1874672518230656/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830246.805415:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518230656/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830246.805422:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830246.805425:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830246.805427:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518230656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830246.805453:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518230784:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830246.805469:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830246.805477:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830246/real 1787830246] req@ffff8b2e88038f00 x1874672518230784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830246.805486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518230784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830246.805494:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830246.805496:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830246.805500:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518230784:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830246.805504:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518230912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830246.805516:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830246.805523:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830246/real 1787830246] req@ffff8b2e8803b840 x1874672518230912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830246.805531:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518230912/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830246.805538:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830246.805540:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830246.805542:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518230912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830246.805548:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518231040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830246.805561:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830246.805570:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830246/real 1787830246] req@ffff8b2e880383c0 x1874672518231040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830246.805580:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518231040/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830271 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830246.805587:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787830246.805588:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787830246.805591:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830246.805597:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518231040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830247.124809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830247.636794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830247.637263:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830247.828784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830247.828849:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830248.148769:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830248.660864:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830248.661018:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830248.852968:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830248.852989:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830249.173389:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830249.685245:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830249.687045:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830249.876785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830249.876900:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830250.197304:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830250.708800:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830250.709205:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830250.901137:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830250.901253:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830251.220790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830251.732769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830251.732817:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830251.924777:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830251.924785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830251.924790:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830251.924795:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830251.924805:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3258 00000100:00080000:3.0:1787830251.924817:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830251.924825:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830251.924878:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518231168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830251.924911:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830251.924914:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830251.924916:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830251.924920:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3258 00000100:00080000:3.0:1787830251.924923:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830251.924927:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830251.924937:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830251.924948:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518231168:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830251.924953:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518231296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830251.924966:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830251.924970:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830251.924972:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830251.924976:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3258 00000100:00080000:3.0:1787830251.924995:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830251.925001:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830251.925006:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000800:00000100:2.0:1787830251.925008:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830251.925022:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518231424/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830251.925033:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830251.925036:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830251.925038:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830251.925042:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3258 00000100:00080000:3.0:1787830251.925046:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830251.925053:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830251.925054:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830251/real 1787830251] req@ffff8b2e880392c0 x1874672518231168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787830251.925071:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830251.925072:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518231168/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830251.925085:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787830251.925089:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518231552/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830251.925091:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830251.925097:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518231168:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830251.925113:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830251.925117:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518231296:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830251.925127:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830251.925130:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830251.925136:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3258 00000800:00000100:2.0:1787830251.925139:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830251.925143:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830251.925150:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830251.925154:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830251/real 1787830251] req@ffff8b2e8803b480 x1874672518231296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787830251.925157:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830251.925171:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518231296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830251.925177:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518231680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830251.925183:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830251.925191:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787830251.925195:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830251.925196:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518231296:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830251.925204:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787830251.925208:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518231424:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830251.925209:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830251.925217:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3258 00000100:00080000:3.0:1787830251.925230:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830251.925236:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:2.0:1787830251.925239:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:3.0:1787830251.925241:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830251.925253:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830251/real 1787830251] req@ffff8b2e8803a1c0 x1874672518231424/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830251.925258:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518231808/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830251.925274:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518231424/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830251.925288:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830251.925293:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787830251.925295:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830251.925298:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518231424:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830251.925301:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830251.925305:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830251.925308:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518231552:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830251.925313:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3258 00000100:00080000:3.0:1787830251.925319:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830251.925323:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:2.0:1787830251.925329:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:3.0:1787830251.925329:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830251.925341:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830251/real 1787830251] req@ffff8b2e88039680 x1874672518231552/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830251.925345:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518231936/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830251.925374:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518231552/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830251.925384:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830251.925387:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830251.925390:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518231552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830251.925405:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518231680:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830251.925421:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830251.925430:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830251/real 1787830251] req@ffff8b2e880392c0 x1874672518231680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830251.925440:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518231680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830251.925448:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830251.925451:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830251.925454:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518231680:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830251.925459:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518231808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830251.925471:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830251.925479:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830251/real 1787830251] req@ffff8b2e880383c0 x1874672518231808/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830251.925489:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518231808/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830251.925496:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830251.925498:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830251.925501:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518231808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830251.925505:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518231936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830251.925517:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830251.925525:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830251/real 1787830251] req@ffff8b2e8803b840 x1874672518231936/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830251.925534:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518231936/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830276 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830251.925542:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830251.925545:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830251.925548:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518231936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830251.926443:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830252.244838:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830252.756859:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830252.756879:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830252.948768:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830252.948842:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830253.268795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830253.780781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830253.780921:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830253.973030:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830253.973041:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830254.293240:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830254.804836:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830254.804854:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830254.996807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830254.996969:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830255.316772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830255.828924:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830255.829271:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830256.020778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830256.020782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830256.341685:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830256.852779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830256.853503:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830257.044790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830257.045080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830257.045085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830257.045089:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830257.045099:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3263 00000100:00080000:3.0:1787830257.045109:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830257.045117:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830257.045155:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518232064/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830257.045176:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830257.045179:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830257.045180:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830257.045183:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3263 00000100:00080000:3.0:1787830257.045185:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830257.045188:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830257.045195:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830257.045204:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518232192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830257.045213:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830257.045216:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830257.045218:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830257.045221:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3263 00000100:00080000:3.0:1787830257.045224:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830257.045227:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830257.045232:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830257.045243:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518232320/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830257.045251:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830257.045253:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830257.045256:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830257.045258:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3263 00000100:00080000:3.0:1787830257.045262:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830257.045265:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830257.045269:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830257.045278:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518232448/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830257.045286:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830257.045289:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830257.045290:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830257.045293:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3263 00000100:00080000:3.0:1787830257.045296:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830257.045299:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830257.045303:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830257.045312:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518232576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830257.045330:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830257.045334:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830257.045336:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830257.045340:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3263 00000100:00080000:3.0:1787830257.045343:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830257.045347:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830257.045353:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830257.045364:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518232704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830257.045373:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830257.045376:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830257.045378:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830257.045381:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3263 00000100:00080000:3.0:1787830257.045386:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830257.045390:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830257.045394:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830257.045404:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518232064:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830257.045408:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518232832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830257.045418:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830257.045509:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518232192:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830257.045536:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518232320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830257.045559:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518232448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830257.045580:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518232576:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830257.045600:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518232704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830257.045643:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518232832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830257.364725:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830257.876909:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830257.877863:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830258.068772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830258.068840:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830258.388819:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830258.900773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830258.900827:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830259.093234:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830259.093794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830259.412909:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830259.924790:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830259.925432:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830260.116980:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830260.117471:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:2.0:1787830260.121584:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:2.0:1787830260.122572:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:2.0:1787830260.122590:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:1.0:1787830260.124951:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830257/real 1787830260] req@ffff8b2e88038f00 x1874672518232064/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830282 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830260.124983:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518232064/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830282 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830260.124995:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830260.125000:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830260.125004:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518232064:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830260.125380:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787830260.125409:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787830260.125418:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787830260.125425:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787830260.125432:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:2.0:1787830260.125439:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:1.0:1787830260.125532:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830257/real 1787830260] req@ffff8b2e880383c0 x1874672518232832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830260.125550:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518232832/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830260.125558:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830260.125562:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830260.125566:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518232832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830260.125572:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830257/real 1787830260] req@ffff8b2e8803a580 x1874672518232192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830260.125585:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518232192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830260.125594:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830260.125597:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830260.125600:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518232192:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830260.125606:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830257/real 1787830260] req@ffff8b2e88039e00 x1874672518232320/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830260.125618:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518232320/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830260.125787:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830260.125793:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830260.125797:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518232320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830260.125805:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830257/real 1787830260] req@ffff8b2e88039a40 x1874672518232448/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830260.126852:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518232448/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830260.126862:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830260.126866:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830260.126870:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518232448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830260.126877:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830257/real 1787830260] req@ffff8b2e8803b0c0 x1874672518232576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830260.126888:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518232576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830260.126895:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830260.126898:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830260.126901:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518232576:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830260.126907:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830257/real 1787830260] req@ffff8b2e8803b840 x1874672518232704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830282 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830260.126918:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518232704/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830282 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830260.126926:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830260.126929:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830260.126933:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518232704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830260.436784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830260.948777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830260.948914:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830261.140808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830261.141631:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830261.460781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830261.973134:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830261.973567:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830262.164806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830262.165724:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830262.165731:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830262.165733:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830262.165741:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3268 00000100:00080000:0.0:1787830262.165749:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830262.165754:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830262.165784:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518232960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830262.165807:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830262.165810:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830262.165811:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830262.165814:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3268 00000100:00080000:0.0:1787830262.165816:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830262.165818:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830262.165826:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830262.165833:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518233088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830262.165839:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830262.165841:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830262.165842:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830262.165844:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3268 00000100:00080000:0.0:1787830262.165845:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830262.165847:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830262.165849:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830262.165854:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518233216/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830262.165859:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830262.165861:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830262.165862:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830262.165864:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3268 00000100:00080000:0.0:1787830262.165866:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830262.165868:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830262.165870:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830262.165874:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518233344/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830262.165878:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830262.165880:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830262.165881:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830262.165883:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3268 00000100:00080000:0.0:1787830262.165885:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830262.165887:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830262.165889:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830262.165894:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518233472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830262.165901:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830262.165904:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830262.165906:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830262.165909:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3268 00000100:00080000:0.0:1787830262.165911:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830262.165914:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830262.165917:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830262.165926:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518233600/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830262.165933:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830262.165936:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830262.165938:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830262.165941:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3268 00000100:00080000:0.0:1787830262.165944:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830262.165947:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830262.165951:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830262.165959:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518233728/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830262.166036:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830262.166104:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518232960:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830262.166158:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830262.166199:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830262/real 1787830262] req@ffff8b2e880421c0 x1874672518232960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830262.166213:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518232960/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830262.166222:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830262.166226:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830262.166229:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518232960:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830262.166235:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518233088:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830262.166255:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830262.166267:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830262/real 1787830262] req@ffff8b2e88041a40 x1874672518233088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830262.166276:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518233088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830262.166301:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830262.166304:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830262.166307:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518233088:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830262.166312:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518233216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830262.166330:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830262.166340:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830262/real 1787830262] req@ffff8b2e88040f00 x1874672518233216/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830262.166350:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518233216/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830262.166357:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830262.166360:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830262.166363:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518233216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830262.166368:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518233344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830262.166386:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830262.166396:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830262/real 1787830262] req@ffff8b2e88042940 x1874672518233344/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830262.166406:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518233344/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830262.166413:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830262.166415:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830262.166419:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518233344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830262.166424:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518233472:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830262.166442:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830262.166451:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830262/real 1787830262] req@ffff8b2e88043480 x1874672518233472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830262.166461:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518233472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830262.166467:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830262.166469:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830262.166471:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518233472:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830262.166477:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518233600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830262.166493:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830262.166501:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830262/real 1787830262] req@ffff8b2e880430c0 x1874672518233600/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830262.166526:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518233600/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830262.166534:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830262.166537:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830262.166540:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518233600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830262.166545:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518233728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830262.166563:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830262.169198:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830262/real 1787830262] req@ffff8b2e88040000 x1874672518233728/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830262.169241:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518233728/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830287 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830262.169250:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830262.169254:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830262.169256:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518233728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830262.484853:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830262.996773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830262.996903:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830263.188809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830263.188860:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830263.509141:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830264.021000:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830264.021696:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830264.212769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830264.212912:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830264.532786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830265.044773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830265.044832:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830265.236800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830265.237076:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830265.556809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830266.068830:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830266.069136:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830266.260790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830266.260894:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830266.580779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830267.093036:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830267.093318:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830267.284780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830267.284784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787830267.284964:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830267.284969:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830267.284973:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830267.284982:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3273 00000100:00080000:2.0:1787830267.284999:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830267.285007:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830267.285046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518233856/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830267.285323:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830267.285326:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830267.285328:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830267.285333:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3273 00000100:00080000:2.0:1787830267.285337:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830267.285340:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830267.285350:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830267.285363:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518233984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830267.285372:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830267.285376:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830267.285377:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830267.285380:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3273 00000100:00080000:2.0:1787830267.285382:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830267.285385:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830267.285390:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830267.285400:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518234112/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830267.285409:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830267.285412:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830267.285413:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830267.285416:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3273 00000100:00080000:2.0:1787830267.285419:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830267.285422:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830267.285426:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830267.285435:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518234240/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830267.285443:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830267.285446:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830267.285448:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830267.285451:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3273 00000100:00080000:2.0:1787830267.285454:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830267.285458:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830267.285462:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830267.285476:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518234368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830267.285486:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830267.285489:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830267.285491:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830267.285495:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3273 00000100:00080000:2.0:1787830267.285499:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830267.285512:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518233856:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787830267.285513:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830267.285522:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830267.285535:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518234496/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830267.285550:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830267.285554:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830267.285556:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830267.285559:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3273 00000100:00080000:2.0:1787830267.285562:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830267.285565:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830267.285569:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830267.285581:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518234624/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830267.285587:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830267.285633:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830267/real 1787830267] req@ffff8b2e8801da40 x1874672518233856/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830267.285680:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518233856/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830267.285690:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830267.285694:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830267.285698:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518233856:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830267.285705:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518233984:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830267.285728:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830267.285740:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830267/real 1787830267] req@ffff8b2e8801fc00 x1874672518233984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830267.285753:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518233984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830267.285761:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830267.285764:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830267.285767:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518233984:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830267.285774:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518234112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830267.285793:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830267.285804:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830267/real 1787830267] req@ffff8b2e8801f480 x1874672518234112/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830267.285815:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518234112/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830267.285822:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830267.285825:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830267.285828:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518234112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830267.285833:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518234240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830267.285851:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830267.285862:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830267/real 1787830267] req@ffff8b2e8801cf00 x1874672518234240/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830267.285878:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518234240/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830267.285886:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830267.285889:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830267.285893:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518234240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830267.285898:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518234368:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830267.285917:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830267.285928:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830267/real 1787830267] req@ffff8b2e8801c000 x1874672518234368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830267.285938:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518234368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830267.285946:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830267.285949:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830267.285952:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518234368:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830267.285978:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518234496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830267.285997:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830267.286007:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830267/real 1787830267] req@ffff8b2e8801e580 x1874672518234496/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830267.286024:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518234496/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830267.286032:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830267.286035:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830267.286038:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518234496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830267.286043:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518234624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830267.286060:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830267.286070:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830267/real 1787830267] req@ffff8b2e8801e940 x1874672518234624/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830267.286080:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518234624/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830292 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830267.286088:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830267.286091:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830267.286094:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518234624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830267.604950:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830268.116815:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830268.120438:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830268.308780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830268.308780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830268.628779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830269.140854:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830269.140989:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830269.332901:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830269.332901:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830269.652811:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830270.164736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830270.164767:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830270.356865:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830270.358625:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830270.678132:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830271.188779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830271.188994:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830271.380781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830271.380795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830271.701147:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830272.212983:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830272.213325:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:2.0:1787830272.404773:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830272.404774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830272.404783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787830272.404784:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830272.404789:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830272.404800:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3278 00000100:00080000:2.0:1787830272.404810:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830272.404817:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830272.404857:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518234752/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830272.404881:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830272.404885:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830272.404887:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830272.404891:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3278 00000100:00080000:2.0:1787830272.404894:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830272.404898:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830272.404908:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830272.404924:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518234752:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830272.404925:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518234880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830272.404939:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830272.404943:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830272.404945:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830272.404948:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3278 00000100:00080000:2.0:1787830272.404952:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830272.404955:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830272.404959:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830272.404969:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518235008/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830272.404978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830272.404981:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830272.404983:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830272.404986:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3278 00000100:00080000:2.0:1787830272.404990:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830272.404993:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830272.404998:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830272.405007:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518235136/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830272.405014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830272.405018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787830272.405021:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787830272.405021:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830272.405028:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3278 00000100:00080000:2.0:1787830272.405031:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830272.405035:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830272.405039:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830272.405048:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518235264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830272.405057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830272.405060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830272.405062:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830272.405068:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3278 00000100:00100000:1.0:1787830272.405069:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830272/real 1787830272] req@ffff8b2e8801e940 x1874672518234752/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830272.405071:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830272.405078:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830272.405086:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830272.405096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518235392/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830272.405103:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830272.405107:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518234752/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830272.405107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830272.405111:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830272.405119:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3278 00000100:00080000:2.0:1787830272.405125:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830272.405127:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830272.405130:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:1.0:1787830272.405135:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:2.0:1787830272.405136:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830272.405142:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518234752:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830272.405151:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518235520/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830272.405169:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518234880:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830272.405195:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830272.405207:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830272/real 1787830272] req@ffff8b2e8801e580 x1874672518234880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830272.405238:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518234880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830272.405246:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830272.405249:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830272.405252:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518234880:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830272.405258:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518235008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830272.405281:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830272.405305:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830272/real 1787830272] req@ffff8b2e8801c000 x1874672518235008/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830272.405326:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518235008/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830272.405334:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830272.405337:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830272.405340:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518235008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830272.405347:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518235136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830272.405366:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830272.405376:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830272/real 1787830272] req@ffff8b2e8801cf00 x1874672518235136/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830272.405387:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518235136/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830272.405394:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830272.405397:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830272.405405:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518235136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830272.405410:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518235264:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830272.405429:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830272.405440:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830272/real 1787830272] req@ffff8b2e8801f480 x1874672518235264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830272.405457:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518235264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830272.405465:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830272.405468:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830272.405471:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518235264:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830272.405476:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518235392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830272.405493:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830272.405502:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830272/real 1787830272] req@ffff8b2e8801fc00 x1874672518235392/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830272.405512:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518235392/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830272.405519:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830272.405533:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830272.405536:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518235392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830272.405542:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518235520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830272.405568:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830272.405579:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830272/real 1787830272] req@ffff8b2e8801da40 x1874672518235520/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830272.405590:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518235520/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830297 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830272.405603:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830272.405606:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830272.405609:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518235520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830272.724797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830273.236885:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830273.237000:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830273.428797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830273.428797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830273.748799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830274.261499:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830274.261609:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830274.452786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830274.452786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830274.772786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830275.284777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830275.284940:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830275.476817:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830275.476862:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830275.796831:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830276.308797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830276.308941:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830276.500819:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830276.500823:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830276.820787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830277.332790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830277.332965:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830277.524787:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830277.524799:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830277.524803:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830277.524813:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3283 00000100:00080000:3.0:1787830277.524824:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830277.524832:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830277.524870:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518235648/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830277.524910:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830277.524913:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830277.524915:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830277.524919:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3283 00000100:00080000:3.0:1787830277.524922:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830277.524925:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830277.524935:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830277.524946:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518235776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830277.524954:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830277.524957:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830277.524959:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830277.524961:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3283 00000100:00080000:3.0:1787830277.524964:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830277.524967:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830277.524971:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830277.524978:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518235904/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830277.524986:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830277.524989:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830277.524991:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830277.524994:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3283 00000100:00080000:3.0:1787830277.524996:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830277.525000:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830277.525004:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830277.525012:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518236032/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830277.525020:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830277.525023:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830277.525024:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830277.525028:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3283 00000100:00080000:3.0:1787830277.525030:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830277.525034:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830277.525037:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830277.525048:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518236160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830277.525056:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830277.525059:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830277.525061:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830277.525063:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3283 00000100:00080000:3.0:1787830277.525066:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830277.525069:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830277.525073:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830277.525081:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518236288/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830277.525093:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830277.525095:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830277.525097:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830277.525100:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3283 00000100:00080000:3.0:1787830277.525103:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830277.525106:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830277.525110:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830277.525119:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518236416/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830277.525410:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518235648:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830277.525468:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830277.525513:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830277/real 1787830277] req@ffff8b2e880392c0 x1874672518235648/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830277.525530:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518235648/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830277.525539:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830277.525543:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830277.525546:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518235648:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830277.525553:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518235776:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830277.525575:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830277.525588:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830277/real 1787830277] req@ffff8b2e88039680 x1874672518235776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830277.525600:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518235776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830277.525608:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830277.525611:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830277.525615:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518235776:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830277.525621:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518235904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830277.525640:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830277.525672:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830277/real 1787830277] req@ffff8b2e8803a1c0 x1874672518235904/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830277.525684:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518235904/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830277.525691:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830277.525694:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830277.525698:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518235904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830277.525703:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518236032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830277.525722:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830277.525733:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830277/real 1787830277] req@ffff8b2e8803b480 x1874672518236032/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830277.525743:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518236032/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830277.525752:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830277.525755:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830277.525758:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518236032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830277.525763:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518236160:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830277.525781:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830277.525791:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830277/real 1787830277] req@ffff8b2e8803b840 x1874672518236160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830277.525801:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518236160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830277.525809:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830277.525812:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830277.525815:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518236160:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830277.525821:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518236288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830277.525839:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830277.525850:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830277/real 1787830277] req@ffff8b2e8803b0c0 x1874672518236288/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830277.525861:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518236288/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830277.525869:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830277.525871:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830277.525875:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518236288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830277.525881:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518236416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830277.525898:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830277.525978:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830277.526360:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830277.527023:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830277/real 1787830277] req@ffff8b2e88039a40 x1874672518236416/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830277.527038:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518236416/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830302 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830277.527046:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830277.527050:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830277.527053:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518236416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830277.844845:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830278.356781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830278.356833:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830278.548759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830278.548759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830278.868963:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830279.380761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830279.381075:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830279.572773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830279.572773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830279.892789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830280.404983:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830280.405071:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830280.596784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830280.597201:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830280.916808:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830281.428784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830281.428946:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830281.620813:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830281.620913:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830281.941137:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830282.452761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830282.453084:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830282.644775:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830282.644780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830282.644788:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830282.644795:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830282.644813:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3289 00000100:00080000:3.0:1787830282.644824:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830282.644833:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830282.644877:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518236544/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830282.644916:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830282.644919:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830282.644922:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830282.644926:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3289 00000100:00080000:3.0:1787830282.644929:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830282.644933:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830282.644943:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830282.644955:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518236672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830282.644965:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830282.644968:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830282.644969:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830282.644973:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3289 00000100:00080000:3.0:1787830282.644977:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830282.644982:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830282.644987:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830282.644992:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518236544:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830282.645003:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518236800/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830282.645014:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830282.645016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830282.645018:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830282.645021:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3289 00000100:00080000:3.0:1787830282.645024:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830282.645028:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830282.645033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830282.645044:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518236928/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830282.645052:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787830282.645055:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830282.645059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830282.645063:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830282.645067:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3289 00000100:00080000:3.0:1787830282.645071:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830282.645075:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830282.645080:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830282.645091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518237056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830282.645100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830282.645105:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830282/real 1787830282] req@ffff8b2e88039e00 x1874672518236544/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830282.645105:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830282.645109:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830282.645117:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3289 00000100:00080000:3.0:1787830282.645123:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830282.645131:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830282.645137:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830282.645153:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518236544/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830282.645162:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518237184/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830282.645166:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830282.645175:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787830282.645181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830282.645182:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518236544:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830282.645189:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830282.645193:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830282.645196:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518236672:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830282.645201:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3289 00000100:00080000:3.0:1787830282.645207:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830282.645212:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830282.645216:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000008:00100000:3.0:1787830282.645218:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:1.0:1787830282.645226:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830282.645235:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518237312/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830282.645243:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830282/real 1787830282] req@ffff8b2e8803a580 x1874672518236672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830282.645257:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518236672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830282.645266:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830282.645269:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830282.645272:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518236672:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830282.645293:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518236800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830282.645309:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830282.645318:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830282/real 1787830282] req@ffff8b2e880383c0 x1874672518236800/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830282.645327:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518236800/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830282.645339:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830282.645342:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830282.645344:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518236800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830282.645349:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518236928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830282.645361:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830282.645369:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830282/real 1787830282] req@ffff8b2e88038f00 x1874672518236928/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830282.645377:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518236928/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830282.645384:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830282.645387:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830282.645389:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518236928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830282.645394:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518237056:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830282.645408:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830282.645416:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830282/real 1787830282] req@ffff8b2e88039a40 x1874672518237056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830282.645427:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518237056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830282.645433:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830282.645436:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830282.645439:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518237056:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830282.645444:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518237184:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830282.645457:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830282.645465:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830282/real 1787830282] req@ffff8b2e8803b0c0 x1874672518237184/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830282.645474:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518237184/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830282.645481:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830282.645483:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830282.645486:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518237184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830282.645490:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518237312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830282.645505:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830282.645514:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830282/real 1787830282] req@ffff8b2e8803b840 x1874672518237312/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830282.645523:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518237312/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830307 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830282.645529:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830282.645532:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830282.645534:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518237312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830282.964933:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830283.476801:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830283.476965:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830283.669375:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830283.669834:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830283.988945:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830284.500897:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830284.500930:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830284.692794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830284.692794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830285.012792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830285.524738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830285.524775:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830285.716735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830285.716756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830286.036767:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830286.548786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830286.548826:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830286.740780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830286.740780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830287.060937:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830287.572785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830287.572930:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830287.764789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830287.764789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787830287.764822:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830287.764829:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830287.764833:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830287.764842:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3294 00000100:00080000:2.0:1787830287.764852:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830287.764860:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830287.764898:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518237440/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830287.764927:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830287.764931:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830287.764933:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830287.764936:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3294 00000100:00080000:2.0:1787830287.764940:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830287.764944:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830287.764953:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830287.764965:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518237568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830287.764974:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830287.764977:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830287.764978:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830287.764981:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3294 00000100:00080000:2.0:1787830287.764984:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830287.764987:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830287.764991:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830287.764999:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518237696/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830287.765006:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830287.765009:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830287.765011:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830287.765014:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3294 00000100:00080000:2.0:1787830287.765017:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830287.765020:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830287.765024:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830287.765032:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518237824/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830287.765038:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830287.765041:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830287.765042:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830287.765045:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3294 00000100:00080000:2.0:1787830287.765048:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830287.765051:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830287.765054:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830287.765062:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518237952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830287.765070:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830287.765073:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830287.765075:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830287.765078:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3294 00000100:00080000:2.0:1787830287.765082:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830287.765085:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830287.765089:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830287.765099:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518238080/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830287.765119:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830287.765122:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830287.765123:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830287.765127:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3294 00000100:00080000:2.0:1787830287.765130:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830287.765133:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830287.765138:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830287.765147:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518238208/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830287.765221:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518237440:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830287.765277:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830287.765326:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830287/real 1787830287] req@ffff8b2e8801ed00 x1874672518237440/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830287.765340:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518237440/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830287.765349:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830287.765353:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830287.765356:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518237440:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830287.765362:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518237568:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830287.765383:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830287.765396:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830287/real 1787830287] req@ffff8b2e8801c780 x1874672518237568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830287.765407:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518237568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830287.765420:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830287.765423:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830287.765427:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518237568:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830287.765432:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518237696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830287.765452:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830287.765463:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830287/real 1787830287] req@ffff8b2e8801f0c0 x1874672518237696/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830287.765475:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518237696/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830287.765484:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830287.765487:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830287.765490:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518237696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830287.765497:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518237824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830287.766175:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830287.766191:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830287/real 1787830287] req@ffff8b2e8801d680 x1874672518237824/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830287.766203:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518237824/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830287.766211:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830287.766214:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830287.766217:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518237824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830287.766224:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518237952:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830287.766244:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830287.766254:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830287/real 1787830287] req@ffff8b2e8801c3c0 x1874672518237952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830287.766264:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518237952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830287.766272:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830287.766274:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830287.766277:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518237952:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830287.766284:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518238080:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830287.766300:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830287.766309:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830287/real 1787830287] req@ffff8b2e8801cb40 x1874672518238080/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830287.766319:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518238080/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830287.766326:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830287.766329:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830287.766332:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518238080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830287.766338:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518238208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830287.766354:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830287.766365:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830287/real 1787830287] req@ffff8b2e8801de00 x1874672518238208/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830287.766375:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518238208/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830312 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830287.766383:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830287.766386:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830287.766388:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518238208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830288.084796:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830288.596775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830288.596820:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830288.788813:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830288.788916:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830289.109180:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830289.620771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830289.620795:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830289.812770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830289.812988:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830290.132890:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830290.644869:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830290.644943:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830290.836732:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830290.837848:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830291.156817:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830291.669161:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830291.669477:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830291.860785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830291.860785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830292.183313:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830292.692779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830292.692976:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830292.884783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830292.884790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830292.884885:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830292.884891:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830292.884895:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830292.884904:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3299 00000100:00080000:3.0:1787830292.884914:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830292.884921:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830292.884960:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518238336/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830292.884990:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830292.884993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830292.884995:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830292.884999:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3299 00000100:00080000:3.0:1787830292.885002:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830292.885006:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830292.885015:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830292.885027:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518238464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830292.885035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830292.885132:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518238336:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830292.885186:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830292.885229:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830292/real 1787830292] req@ffff8b2e8803b480 x1874672518238336/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830292.885244:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518238336/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830292.885253:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830292.885257:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830292.885260:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518238336:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830292.885267:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518238464:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830292.885288:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830292.885301:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830292/real 1787830292] req@ffff8b2e8803a1c0 x1874672518238464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830292.885312:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518238464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830292.885320:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830292.885323:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830292.885326:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518238464:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830292.885375:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830292.885377:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830292.885381:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3299 00000100:00080000:3.0:1787830292.885385:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830292.885389:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830292.885395:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830292.885406:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518238592/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830292.885424:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830292.885428:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830292.885429:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830292.885432:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3299 00000100:00080000:3.0:1787830292.885436:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830292.885438:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830292.885442:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830292.885452:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518238720/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830292.885458:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830292.885461:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830292.885462:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830292.885465:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3299 00000100:00080000:3.0:1787830292.885468:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830292.885471:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830292.885474:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830292.885483:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518238848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830292.885489:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830292.885491:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830292.885493:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830292.885496:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3299 00000100:00080000:3.0:1787830292.885498:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830292.885501:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830292.885504:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830292.885511:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518238976/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830292.885519:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830292.885521:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830292.885522:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830292.885525:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3299 00000100:00080000:3.0:1787830292.885539:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830292.885543:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830292.885547:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830292.885560:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518239104/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830292.885947:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518238592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830292.885976:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830292.885991:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830292/real 1787830292] req@ffff8b2e88039680 x1874672518238592/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830292.886005:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518238592/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830292.886015:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830292.886018:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830292.886022:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518238592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830292.886028:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518238720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830292.886048:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830292.886059:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830292/real 1787830292] req@ffff8b2e880392c0 x1874672518238720/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830292.886138:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518238720/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830292.886148:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830292.886151:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830292.886155:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518238720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830292.886161:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518238848:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830292.886189:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830292.886201:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830292/real 1787830292] req@ffff8b2e8803a1c0 x1874672518238848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830292.886213:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518238848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830292.886222:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830292.886226:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830292.886229:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518238848:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830292.886235:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518238976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830292.886254:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830292.886265:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830292/real 1787830292] req@ffff8b2e8803b480 x1874672518238976/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830292.886284:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518238976/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830292.886292:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830292.886295:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830292.886298:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518238976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830292.886304:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518239104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830292.886323:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830292.886334:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830292/real 1787830292] req@ffff8b2e8803b840 x1874672518239104/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830292.886344:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518239104/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830317 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830292.886352:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830292.886355:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830292.886358:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518239104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830293.204951:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830293.716774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830293.717097:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830293.908792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830293.908792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830294.229140:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830294.740921:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830294.740938:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830294.932793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830294.932880:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830295.252791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830295.764760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830295.764835:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830295.956778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830295.956779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830296.276801:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830296.793754:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830296.793836:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830296.980785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830296.980873:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830297.300789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830297.812788:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830297.813183:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830298.004793:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830298.004796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830298.004797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830298.004806:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830298.004811:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830298.004820:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3304 00000100:00080000:3.0:1787830298.004831:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830298.004838:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830298.004879:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518239232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830298.004907:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830298.004916:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830298.004917:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830298.004921:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3304 00000100:00080000:3.0:1787830298.004924:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830298.004927:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830298.004936:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830298.004946:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518239360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830298.004954:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830298.004957:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830298.004958:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830298.004961:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3304 00000100:00080000:3.0:1787830298.004963:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830298.004967:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830298.004971:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830298.004981:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518239488/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830298.004988:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830298.004991:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830298.004993:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830298.004995:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3304 00000100:00080000:3.0:1787830298.004997:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830298.005001:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830298.005004:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830298.005011:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518239616/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830298.005018:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830298.005020:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830298.005022:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830298.005025:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3304 00000100:00080000:3.0:1787830298.005027:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830298.005030:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830298.005033:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830298.005062:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518239744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830298.005070:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830298.005072:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830298.005074:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830298.005077:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3304 00000100:00080000:3.0:1787830298.005079:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830298.005083:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830298.005087:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830298.005094:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518239872/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830298.005101:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830298.005103:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830298.005104:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830298.005107:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3304 00000100:00080000:3.0:1787830298.005109:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830298.005112:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830298.005115:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830298.005122:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518240000/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830298.005681:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518239232:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830298.005738:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830298.005780:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830298/real 1787830298] req@ffff8b2e8803b0c0 x1874672518239232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830298.005806:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518239232/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830298.005817:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830298.005821:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830298.005824:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518239232:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830298.005831:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518239360:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830298.005852:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830298.005864:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830298/real 1787830298] req@ffff8b2e88039a40 x1874672518239360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830298.005877:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518239360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830298.005886:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830298.005889:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830298.005892:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518239360:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830298.005898:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518239488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830298.005919:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830298.005930:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830298/real 1787830298] req@ffff8b2e88038f00 x1874672518239488/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830298.005941:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518239488/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830298.005949:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830298.005963:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830298.005966:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518239488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830298.005973:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518239616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830298.005989:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830298.005999:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830298/real 1787830298] req@ffff8b2e880383c0 x1874672518239616/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830298.006010:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518239616/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830298.006018:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830298.006021:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830298.006028:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518239616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830298.006033:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518239744:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830298.006061:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830298.006071:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830298/real 1787830298] req@ffff8b2e8803a580 x1874672518239744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830298.006088:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518239744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830298.006096:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830298.006099:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830298.006102:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518239744:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830298.006108:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518239872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830298.006124:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830298.006131:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830298/real 1787830298] req@ffff8b2e88039e00 x1874672518239872/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830298.006137:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518239872/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830298.006142:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830298.006144:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830298.006146:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518239872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830298.006149:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518240000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830298.006166:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830298.006172:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830298/real 1787830298] req@ffff8b2e8803b840 x1874672518240000/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830298.006178:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518240000/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830323 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830298.006191:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830298.006194:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830298.006196:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518240000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830298.325086:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830298.837067:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830298.837261:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830299.028793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830299.030768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830299.349136:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830299.862217:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830299.862237:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830300.052771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830300.052771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830300.373857:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830300.886188:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830300.886695:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830301.076944:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830301.077056:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830301.396836:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830301.908832:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830301.908875:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830302.100761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830302.100761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830302.420887:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830302.932871:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830302.934676:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830303.124759:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830303.124761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830303.124761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830303.124771:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830303.124776:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830303.124786:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3309 00000100:00080000:3.0:1787830303.125214:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830303.125224:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830303.125262:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518240128/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830303.125293:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830303.125297:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830303.125298:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830303.125303:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3309 00000100:00080000:3.0:1787830303.125307:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830303.125311:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830303.125320:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830303.125332:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518240256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830303.125342:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830303.125345:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830303.125347:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830303.125350:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3309 00000100:00080000:3.0:1787830303.125353:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830303.125357:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830303.125361:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830303.125371:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518240384/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830303.125380:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830303.125383:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830303.125385:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830303.125388:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3309 00000100:00080000:3.0:1787830303.125390:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830303.125393:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830303.125398:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830303.125408:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518240512/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830303.125417:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830303.125420:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830303.125421:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830303.125425:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3309 00000100:00080000:3.0:1787830303.125428:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830303.125432:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830303.125436:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830303.125447:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518240640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830303.125456:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830303.125459:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830303.125461:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830303.125464:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3309 00000100:00080000:3.0:1787830303.125467:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830303.125470:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830303.125474:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830303.125484:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518240768/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830303.125491:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830303.125494:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830303.125496:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830303.125499:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3309 00000100:00080000:3.0:1787830303.125502:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830303.125505:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830303.125509:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830303.125517:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518240896/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830303.125638:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518240128:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830303.125705:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830303.125747:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830303/real 1787830303] req@ffff8b2e8803b480 x1874672518240128/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830303.125772:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518240128/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830303.125782:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830303.125786:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830303.125789:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518240128:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830303.125795:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518240256:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830303.125817:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830303.125828:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830303/real 1787830303] req@ffff8b2e8803a1c0 x1874672518240256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830303.125840:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518240256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830303.125847:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830303.125850:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830303.125853:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518240256:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830303.125859:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518240384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830303.125878:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830303.125888:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830303/real 1787830303] req@ffff8b2e880392c0 x1874672518240384/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830303.125899:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518240384/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830303.125912:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830303.125915:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830303.125918:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518240384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830303.125925:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518240512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830303.125944:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830303.125955:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830303/real 1787830303] req@ffff8b2e88039680 x1874672518240512/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830303.125973:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518240512/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830303.125980:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830303.125983:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830303.125986:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518240512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830303.125991:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518240640:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830303.126338:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830303.126354:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830303/real 1787830303] req@ffff8b2e8803b840 x1874672518240640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830303.126364:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518240640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830303.126371:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830303.126374:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830303.126377:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518240640:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830303.126382:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518240768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830303.126401:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830303.126423:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830303/real 1787830303] req@ffff8b2e88039e00 x1874672518240768/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830303.126432:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518240768/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830303.126445:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830303.126448:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830303.126451:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518240768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830303.126456:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518240896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830303.126474:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830303.126484:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830303/real 1787830303] req@ffff8b2e8803a580 x1874672518240896/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830303.126494:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518240896/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830328 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830303.126501:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830303.126504:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830303.126507:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518240896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830303.444792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830303.956796:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830303.956901:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830304.148967:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830304.150375:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830304.469055:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830304.980761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830304.980815:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830305.172871:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830305.172877:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830305.492977:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830306.005598:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830306.005746:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830306.196784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830306.197196:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830306.516794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830307.028780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830307.029053:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830307.220833:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830307.221026:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830307.540784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830308.052796:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830308.052800:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830308.244781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830308.244786:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830308.244794:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830308.244798:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830308.244807:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3314 00000100:00080000:3.0:1787830308.244818:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830308.244825:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830308.244864:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518241024/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830308.244896:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830308.244900:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830308.244901:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830308.244906:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3314 00000100:00080000:3.0:1787830308.244910:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830308.244913:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830308.244923:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830308.244934:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518241152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830308.244943:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830308.244946:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830308.244948:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830308.244953:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3314 00000100:00080000:3.0:1787830308.244956:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830308.244960:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830308.244974:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830308.244989:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518241280/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830308.244990:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830308.244998:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830308.245001:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830308.245003:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830308.245006:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3314 00000100:00080000:3.0:1787830308.245009:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830308.245013:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830308.245017:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830308.245027:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518241408/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830308.245035:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830308.245038:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518241024:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830308.245039:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830308.245041:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830308.245049:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3314 00000100:00080000:3.0:1787830308.245101:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787830308.245106:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830308.245111:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830308.245117:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830308.245130:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518241536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830308.245139:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830308.245142:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830308.245144:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830308.245148:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3314 00000100:00080000:3.0:1787830308.245152:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830308.245157:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830308/real 1787830308] req@ffff8b2e880383c0 x1874672518241024/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830308.245160:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830308.245168:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830308.245182:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518241024/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830308.245183:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518241664/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830308.245212:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830308.245213:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830308.245217:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830308.245222:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787830308.245222:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830308.245231:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518241024:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830308.245231:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3314 00000100:00080000:3.0:1787830308.245238:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830308.245244:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518241152:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830308.245245:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830308.245253:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830308.245267:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518241792/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830308.245277:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830308.245293:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830308/real 1787830308] req@ffff8b2e88038f00 x1874672518241152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830308.245306:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518241152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830308.245314:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830308.245317:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830308.245320:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518241152:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830308.245342:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518241280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830308.245360:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830308.245370:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830308/real 1787830308] req@ffff8b2e88039a40 x1874672518241280/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830308.245387:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518241280/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830308.245394:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830308.245398:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830308.245401:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518241280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830308.245407:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518241408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830308.245422:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830308.245432:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830308/real 1787830308] req@ffff8b2e8803b0c0 x1874672518241408/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830308.245442:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518241408/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830308.245449:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830308.245453:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830308.245456:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518241408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830308.245461:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518241536:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830308.245479:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830308.245489:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830308/real 1787830308] req@ffff8b2e8803a580 x1874672518241536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830308.245500:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518241536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830308.245513:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830308.245516:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830308.245519:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518241536:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830308.245525:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518241664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830308.245541:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830308.245551:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830308/real 1787830308] req@ffff8b2e88039e00 x1874672518241664/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830308.245561:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518241664/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830308.245568:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830308.245571:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830308.245574:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518241664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830308.245579:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518241792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830308.245596:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830308.245606:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830308/real 1787830308] req@ffff8b2e8803b840 x1874672518241792/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830308.245616:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518241792/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830333 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830308.245623:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830308.245626:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830308.245629:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518241792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830308.564792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830309.076782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830309.076989:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830309.268808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830309.268831:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830309.588752:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830310.100774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830310.100987:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830310.293180:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830310.293843:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830310.612856:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830311.124783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830311.124883:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830311.316774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830311.317401:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830311.639158:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830312.148777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830312.148874:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830312.340759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830312.340759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830312.661030:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830313.172781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830313.173338:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830313.364817:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830313.364818:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830313.364828:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830313.364834:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830313.364845:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830313.364847:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3319 00000100:00080000:1.0:1787830313.364875:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830313.364888:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830313.364945:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518241920/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830313.364981:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830313.364985:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830313.364987:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830313.364991:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3319 00000100:00080000:1.0:1787830313.364995:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830313.365000:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830313.365012:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830313.365027:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518242048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830313.365044:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830313.365048:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830313.365050:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830313.365054:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3319 00000100:00080000:1.0:1787830313.365058:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830313.365063:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830313.365069:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830313.365084:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518242176/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830313.365093:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518241920:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830313.365102:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830313.365107:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830313.365109:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830313.365114:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3319 00000100:00080000:1.0:1787830313.365119:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830313.365124:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830313.365130:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830313.365145:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518242304/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830313.365155:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830313.365159:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830313.365162:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787830313.365172:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830313.365177:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3319 00000100:00080000:1.0:1787830313.365185:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830313.365191:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830313.365197:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830313.365212:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518242432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830313.365222:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830313.365228:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:2.0:1787830313.365230:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830313/real 1787830313] req@ffff8b2e88076580 x1874672518241920/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830313.365231:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830313.365242:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3319 00000100:00080000:1.0:1787830313.365251:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830313.365260:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830313.365266:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830313.365269:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518241920/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830313.365284:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787830313.365285:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518242560/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830313.365297:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830313.365310:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518241920:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830313.365313:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830313.365321:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830313.365323:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830313.365329:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518242048:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830313.365332:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3319 00000100:00080000:1.0:1787830313.365341:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830313.365347:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830313.365353:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787830313.365370:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830313.365371:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518242688/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830313.365388:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830313/real 1787830313] req@ffff8b2e88075e00 x1874672518242048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830313.365411:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518242048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830313.365420:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830313.365423:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830313.365426:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518242048:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830313.365451:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518242176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830313.365472:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830313.365482:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830313/real 1787830313] req@ffff8b2e88074b40 x1874672518242176/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830313.365493:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518242176/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830313.365500:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830313.365503:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830313.365506:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518242176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830313.365511:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518242304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830313.365529:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830313.365538:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830313/real 1787830313] req@ffff8b2e88074000 x1874672518242304/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830313.365547:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518242304/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830313.365559:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830313.365562:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830313.365564:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518242304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830313.365570:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518242432:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830313.366250:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830313.366264:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830313/real 1787830313] req@ffff8b2e880761c0 x1874672518242432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830313.366276:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518242432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830313.366284:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830313.366287:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830313.366290:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518242432:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830313.366297:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518242560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830313.366318:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830313.366329:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830313/real 1787830313] req@ffff8b2e880770c0 x1874672518242560/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830313.366340:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518242560/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830313.366347:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830313.366350:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830313.366353:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518242560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830313.366359:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518242688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830313.366376:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830313.366386:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830313/real 1787830313] req@ffff8b2e88077840 x1874672518242688/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830313.366395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518242688/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830338 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830313.366402:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830313.366405:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830313.366408:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518242688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830313.684788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830314.196839:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830314.197169:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830314.389852:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830314.389972:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830314.709456:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830315.221717:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830315.223178:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830315.412793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830315.413379:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830315.733905:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830316.244997:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830316.245783:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830316.436795:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830316.437108:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830316.757449:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830317.268777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830317.268859:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830317.460761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830317.460761:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830317.780781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830318.292780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830318.294140:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830318.484801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830318.484871:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830318.484960:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830318.484965:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830318.484970:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830318.484980:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3324 00000100:00080000:3.0:1787830318.484991:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830318.484998:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830318.485041:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518242816/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830318.485073:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830318.485077:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830318.485078:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830318.485082:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3324 00000100:00080000:3.0:1787830318.485087:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830318.485091:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830318.485101:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830318.485124:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518242944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830318.485134:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830318.485137:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830318.485139:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830318.485142:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3324 00000100:00080000:3.0:1787830318.485369:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830318.485376:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830318.485381:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830318.485393:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518243072/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830318.485403:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830318.485406:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830318.485408:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830318.485411:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3324 00000100:00080000:3.0:1787830318.485415:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830318.485419:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830318.485424:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830318.485434:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518243200/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830318.485442:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830318.485455:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830318.485457:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830318.485462:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3324 00000100:00080000:3.0:1787830318.485465:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830318.485469:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830318.485473:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830318.485485:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518243328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830318.485495:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830318.485498:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830318.485499:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830318.485502:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3324 00000100:00080000:3.0:1787830318.485505:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830318.485508:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830318.485512:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830318.485522:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518243456/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830318.485531:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830318.485534:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830318.485535:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830318.485538:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3324 00000100:00080000:3.0:1787830318.485541:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830318.485544:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830318.485547:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830318.485556:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518243584/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830318.485693:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518242816:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830318.485752:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830318.485799:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830318/real 1787830318] req@ffff8b2e88039680 x1874672518242816/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830318.485822:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518242816/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830318.485832:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830318.485836:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830318.485839:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518242816:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830318.485846:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518242944:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830318.485870:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830318.485883:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830318/real 1787830318] req@ffff8b2e880392c0 x1874672518242944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830318.485904:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518242944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830318.485912:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830318.485915:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830318.485918:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518242944:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830318.485923:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518243072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830318.485942:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830318.485953:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830318/real 1787830318] req@ffff8b2e8803a1c0 x1874672518243072/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830318.485964:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518243072/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830318.485974:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830318.485977:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830318.485980:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518243072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830318.485986:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518243200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830318.486003:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830318.486013:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830318/real 1787830318] req@ffff8b2e8803b480 x1874672518243200/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830318.486024:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518243200/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830318.486031:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830318.486034:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830318.486036:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518243200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830318.486042:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518243328:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830318.486061:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830318.486071:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830318/real 1787830318] req@ffff8b2e8803b840 x1874672518243328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830318.486082:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518243328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830318.486089:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830318.486092:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830318.486096:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518243328:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830318.486101:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518243456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830318.486118:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830318.486127:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830318/real 1787830318] req@ffff8b2e88039e00 x1874672518243456/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830318.486136:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518243456/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830318.486142:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830318.486144:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830318.486438:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518243456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830318.486449:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518243584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830318.486475:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830318.486487:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830318/real 1787830318] req@ffff8b2e8803a580 x1874672518243584/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830318.486498:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518243584/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830343 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830318.486506:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830318.486509:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830318.486512:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518243584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830318.804864:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830319.317034:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830319.321217:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830319.508860:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830319.508940:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830319.828955:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830320.341393:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830320.341679:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830320.532759:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830320.532759:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830320.854006:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830321.364919:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830321.364944:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830321.556772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830321.560430:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830321.876854:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830322.388881:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830322.388932:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830322.580772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830322.580785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830322.900925:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830323.413165:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830323.413349:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830323.605290:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830323.606917:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830323.610220:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830323.610232:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830323.610236:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830323.610246:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3330 00000100:00080000:1.0:1787830323.610257:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830323.610264:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830323.610303:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518243712/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830323.610345:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830323.610350:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830323.610352:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830323.610356:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3330 00000100:00080000:1.0:1787830323.610359:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830323.610362:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830323.610371:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830323.610382:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518243840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830323.610392:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830323.610395:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830323.610396:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830323.610400:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3330 00000100:00080000:1.0:1787830323.610403:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830323.610406:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830323.610410:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830323.610420:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518243968/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830323.610423:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518243712:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830323.610428:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830323.610431:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830323.610432:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830323.610435:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3330 00000100:00080000:1.0:1787830323.610438:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830323.610441:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830323.610444:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830323.610452:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518244096/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830323.610458:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830323.610461:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830323.610462:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830323.610465:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3330 00000100:00080000:1.0:1787830323.610469:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830323.610472:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830323.610475:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830323.610482:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518244224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830323.610489:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830323.610492:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830323.610494:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830323.610497:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3330 00000100:00080000:1.0:1787830323.610499:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830323.610502:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830323.610505:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830323.610514:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518244352/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830323.610520:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518243840:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830323.610521:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830323.610524:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830323.610525:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830323.610528:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3330 00000100:00080000:1.0:1787830323.610531:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830323.610534:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830323.610538:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830323.610546:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518244480/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830323.610549:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518243968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830323.610570:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518244096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830323.610586:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518244224:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830323.610606:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518244352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830323.610626:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518244480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830323.925765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830324.436972:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830324.437018:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830324.628808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830324.628909:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830324.948766:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830325.460804:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830325.460907:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830325.652789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830325.653035:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830325.972819:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830326.484810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830326.485626:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830326.676880:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830326.676880:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:3.0:1787830326.677464:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:3.0:1787830326.677581:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787830326.677638:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830326.677779:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830326.677788:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830326.677796:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830326.677814:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830326.677821:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830326.677850:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:2.0:1787830326.679125:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830323/real 1787830326] req@ffff8b2e88077480 x1874672518244480/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830326.679156:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518244480/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830326.679173:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830326.679178:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830326.679182:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518244480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830326.679189:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830323/real 1787830326] req@ffff8b2e88075680 x1874672518243968/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830326.679203:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518243968/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830326.679211:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830326.679214:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830326.679218:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518243968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830326.679224:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830323/real 1787830326] req@ffff8b2e88074780 x1874672518244096/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830326.679236:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518244096/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830326.679243:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830326.679246:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830326.679249:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518244096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830326.679255:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830323/real 1787830326] req@ffff8b2e88074f00 x1874672518244224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830326.679266:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518244224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830326.679274:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830326.679277:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830326.679280:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518244224:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830326.679286:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830323/real 1787830326] req@ffff8b2e88075a40 x1874672518244352/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830326.679296:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518244352/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830326.679303:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830326.679306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830326.679309:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518244352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830326.679315:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830323/real 1787830326] req@ffff8b2e88076d00 x1874672518243712/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830326.679332:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518243712/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830326.679340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830326.679343:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830326.679346:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518243712:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830326.679352:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830323/real 1787830326] req@ffff8b2e88077c00 x1874672518243840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830326.679363:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518243840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830348 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830326.679371:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830326.679374:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830326.679377:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518243840:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830326.996738:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830327.508804:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830327.508840:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830327.700801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830327.700897:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830328.021086:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830328.532770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830328.532905:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830328.724783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830328.724799:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830328.724806:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830328.724810:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830328.724819:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3335 00000100:00080000:1.0:1787830328.724841:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830328.724850:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830328.724883:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518244608/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830328.724907:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830328.724913:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830328.724915:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830328.724919:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3335 00000100:00080000:1.0:1787830328.724923:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830328.724927:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830328.724936:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830328.724949:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518244736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830328.724960:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830328.724963:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830328.724965:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830328.724968:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3335 00000100:00080000:1.0:1787830328.724971:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830328.724975:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830328.724979:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830328.725521:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518244864/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830328.725533:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830328.725536:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830328.725538:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830328.725542:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3335 00000100:00080000:1.0:1787830328.725546:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830328.725550:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830328.725556:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830328.725568:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518244992/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830328.725578:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830328.725581:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830328.725583:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830328.725586:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3335 00000100:00080000:1.0:1787830328.725589:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830328.725593:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830328.725597:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830328.725607:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518245120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830328.725615:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830328.725618:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830328.725620:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830328.725623:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3335 00000100:00080000:1.0:1787830328.725626:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830328.725629:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830328.725633:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830328.725642:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518245248/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830328.725676:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830328.725678:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830328.725680:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830328.725684:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3335 00000100:00080000:1.0:1787830328.725688:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830328.725691:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830328.725696:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830328.725706:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518245376/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830328.726975:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518244608:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830328.727049:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830328.727099:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830328/real 1787830328] req@ffff8b2e88076940 x1874672518244608/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830328.727116:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518244608/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830328.727127:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830328.727130:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830328.727134:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518244608:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830328.727142:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518244736:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830328.727163:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830328.727175:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830328/real 1787830328] req@ffff8b2e88077c00 x1874672518244736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830328.727185:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518244736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830328.727198:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830328.727201:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830328.727204:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518244736:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830328.727209:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518244864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830328.727228:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830328.727237:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830328/real 1787830328] req@ffff8b2e88076d00 x1874672518244864/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830328.727246:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518244864/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830328.727252:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830328.727254:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830328.727257:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518244864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830328.727262:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518244992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830328.727276:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830328.727284:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830328/real 1787830328] req@ffff8b2e88075a40 x1874672518244992/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830328.727293:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518244992/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830328.727299:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830328.727302:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830328.727306:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518244992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830328.727311:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518245120:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830328.727327:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830328.727337:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830328/real 1787830328] req@ffff8b2e88074f00 x1874672518245120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830328.727347:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518245120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830328.727355:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830328.727357:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830328.727360:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518245120:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830328.727365:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518245248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830328.727380:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830328.727389:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830328/real 1787830328] req@ffff8b2e88074780 x1874672518245248/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830328.727399:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518245248/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830328.727406:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830328.727409:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830328.727412:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518245248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830328.727417:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518245376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830328.727432:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830328.727442:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830328/real 1787830328] req@ffff8b2e88075680 x1874672518245376/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830328.727451:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518245376/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830353 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830328.727457:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830328.727460:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830328.727463:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518245376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830328.727668:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830329.045962:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830329.556762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830329.556846:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830329.748912:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830329.748962:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830330.069140:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830330.580780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830330.580994:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830330.773513:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830330.773882:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830331.092793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830331.605001:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830331.605137:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830331.796822:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830331.796918:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830332.117595:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830332.629146:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830332.629165:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830332.820765:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830332.820765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830333.140808:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830333.652839:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830333.653570:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830333.844895:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830333.845090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830333.845096:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830333.845099:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830333.845108:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3340 00000100:00080000:3.0:1787830333.845117:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830333.845123:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830333.845227:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518245504/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830333.845250:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830333.845254:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830333.845255:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830333.845260:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3340 00000100:00080000:3.0:1787830333.845264:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830333.845267:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830333.845276:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830333.845287:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518245632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830333.845296:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830333.845299:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830333.845300:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830333.845304:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3340 00000100:00080000:3.0:1787830333.845306:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830333.845310:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830333.845314:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830333.845324:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518245760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830333.845333:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830333.845335:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830333.845337:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830333.845340:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3340 00000100:00080000:3.0:1787830333.845343:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830333.845346:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830333.845350:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830333.845357:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518245888/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830333.845365:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830333.845368:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830333.845370:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830333.845373:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3340 00000100:00080000:3.0:1787830333.845375:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830333.845379:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830333.845383:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830333.845392:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518246016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830333.845400:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830333.845403:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830333.845405:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830333.845408:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3340 00000100:00080000:3.0:1787830333.845411:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830333.845426:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830333.845431:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830333.845440:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518246144/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830333.845448:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830333.845451:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830333.845452:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830333.845455:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3340 00000100:00080000:3.0:1787830333.845458:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830333.845462:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830333.845465:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830333.845474:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518246272/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830333.845611:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518245504:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830333.845701:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830333.845748:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830333/real 1787830333] req@ffff8b2e8803b0c0 x1874672518245504/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830333.845763:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518245504/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830333.845773:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830333.845777:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830333.845781:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518245504:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830333.845799:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518245632:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830333.845822:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830333.845835:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830333/real 1787830333] req@ffff8b2e88039a40 x1874672518245632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830333.845846:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518245632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830333.845855:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830333.845858:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830333.845861:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518245632:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830333.845867:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518245760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830333.845887:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830333.845898:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830333/real 1787830333] req@ffff8b2e88038f00 x1874672518245760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830333.845909:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518245760/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830333.845917:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830333.845920:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830333.845923:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518245760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830333.845929:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518245888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830333.845947:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830333.845959:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830333/real 1787830333] req@ffff8b2e880383c0 x1874672518245888/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830333.845971:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518245888/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830333.845978:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830333.845981:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830333.845985:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518245888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830333.845990:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518246016:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830333.846009:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830333.846021:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830333/real 1787830333] req@ffff8b2e8803a580 x1874672518246016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830333.846033:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518246016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830333.846040:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830333.846043:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830333.846046:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518246016:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830333.846052:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518246144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830333.846070:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830333.846081:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830333/real 1787830333] req@ffff8b2e88039e00 x1874672518246144/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830333.846093:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518246144/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830333.846107:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830333.846110:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830333.846114:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518246144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830333.846119:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518246272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830333.846136:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830333.846147:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830333/real 1787830333] req@ffff8b2e8803b840 x1874672518246272/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830333.846158:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518246272/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830358 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830333.846166:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830333.846169:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830333.846172:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518246272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830333.846277:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830334.164876:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830334.676996:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830334.681115:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830334.868753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830334.868870:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830335.188810:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830335.700798:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830335.701022:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830335.892785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830335.892786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830336.212802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830336.725007:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830336.725081:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830336.916804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830336.916805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830337.236787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830337.748842:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830337.749684:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830337.940877:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830337.941007:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830338.260839:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830338.772857:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830338.773194:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830338.964836:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830338.964952:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830338.964958:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830338.964962:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830338.964972:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3345 00000100:00080000:3.0:1787830338.964982:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830338.964990:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830338.965028:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518246400/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830338.965061:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830338.965065:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830338.965067:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830338.965071:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3345 00000100:00080000:3.0:1787830338.965075:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830338.965080:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830338.965091:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830338.965103:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518246528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830338.965112:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830338.965116:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830338.965118:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830338.965122:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3345 00000100:00080000:3.0:1787830338.965125:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830338.965130:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830338.965135:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830338.965145:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518246656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830338.965152:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518246400:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830338.965158:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830338.965163:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830338.965165:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830338.965169:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3345 00000100:00080000:3.0:1787830338.965173:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830338.965177:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830338.965181:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830338.965206:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518246784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830338.965217:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787830338.965222:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830338.965224:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830338.965228:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830338.965234:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3345 00000100:00080000:3.0:1787830338.965238:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830338.965242:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830338.965249:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830338.965264:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518246912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830338.965274:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830338/real 1787830338] req@ffff8b2e8803b480 x1874672518246400/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830338.965275:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830338.965284:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830338.965289:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830338.965297:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3345 00000100:00080000:3.0:1787830338.965301:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830338.965304:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518246400/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830338.965308:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830338.965316:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787830338.965322:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830338.965328:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830338.965331:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518247040/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830338.965336:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518246400:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830338.965350:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518246528:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830338.965351:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830338.965367:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830338.965369:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830338.965374:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3345 00000100:00080000:3.0:1787830338.965379:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830338.965382:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830338.965387:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830338.965397:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518247168/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830338.965514:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830338.965531:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830338/real 1787830338] req@ffff8b2e8803a1c0 x1874672518246528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830338.965545:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518246528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830338.965554:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830338.965558:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830338.965561:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518246528:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830338.965584:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518246656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830338.965608:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830338.965621:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830338/real 1787830338] req@ffff8b2e880392c0 x1874672518246656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830338.965632:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518246656/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830338.965641:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830338.965644:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830338.965670:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518246656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830338.965677:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518246784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830338.965698:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830338.965710:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830338/real 1787830338] req@ffff8b2e88039680 x1874672518246784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830338.965722:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518246784/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830338.965730:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830338.965733:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830338.965736:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518246784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830338.965742:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518246912:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830338.965777:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830338.965788:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830338/real 1787830338] req@ffff8b2e8803b840 x1874672518246912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830338.965799:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518246912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830338.965807:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830338.965810:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830338.965813:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518246912:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830338.965818:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518247040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830338.965838:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830338.965850:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830338/real 1787830338] req@ffff8b2e88039e00 x1874672518247040/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830338.965861:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518247040/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830338.965868:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830338.965871:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830338.965874:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518247040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830338.965879:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518247168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830338.965898:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830338.965910:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830338/real 1787830338] req@ffff8b2e8803a580 x1874672518247168/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830338.965921:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518247168/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830363 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830338.965929:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830338.965932:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830338.965935:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518247168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830338.966305:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830339.284800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830339.796781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830339.796809:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830339.988805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830339.989262:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830340.309173:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830340.821664:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830340.821665:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830341.012786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830341.012787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830341.332901:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830341.844812:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830341.844919:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830342.036808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830342.036921:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830342.356816:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830342.868840:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830342.869440:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830343.060808:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830343.061498:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830343.380827:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830343.893417:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830343.893431:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830344.084793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830344.084827:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830344.084839:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830344.084843:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830344.084854:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3350 00000100:00080000:1.0:1787830344.084864:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830344.084873:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830344.084906:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830344.084913:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518247296/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830344.084939:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830344.084943:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830344.084945:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830344.084949:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3350 00000100:00080000:1.0:1787830344.084953:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830344.084956:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830344.084967:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830344.084978:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518247424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830344.084996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830344.084999:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830344.085001:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830344.085005:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3350 00000100:00080000:1.0:1787830344.085008:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830344.085012:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830344.085017:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830344.085027:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518247552/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830344.085036:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830344.085039:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830344.085042:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830344.085045:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3350 00000100:00080000:1.0:1787830344.085049:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830344.085052:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830344.085056:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830344.085068:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518247680/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830344.085077:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830344.085080:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830344.085082:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830344.085087:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3350 00000100:00080000:1.0:1787830344.085090:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830344.085094:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830344.085099:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830344.085107:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518247808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830344.085116:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830344.085119:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830344.085121:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830344.085124:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3350 00000100:00080000:1.0:1787830344.085127:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830344.085130:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830344.085134:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830344.085143:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518247936/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830344.085152:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830344.085155:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830344.085157:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830344.085160:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3350 00000100:00080000:1.0:1787830344.085164:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830344.085168:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830344.085171:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830344.085180:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518248064/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830344.085223:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518247296:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830344.085283:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830344.085334:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830344/real 1787830344] req@ffff8b2e88077480 x1874672518247296/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830344.085351:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518247296/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830344.085360:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830344.085364:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830344.085368:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518247296:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830344.085375:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518247424:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830344.085404:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830344.085413:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830344/real 1787830344] req@ffff8b2e88077840 x1874672518247424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830344.085421:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518247424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830344.085426:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830344.085428:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830344.085430:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518247424:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830344.085434:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518247552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830344.085448:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830344.085470:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830344/real 1787830344] req@ffff8b2e880770c0 x1874672518247552/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830344.085494:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518247552/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830344.085501:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830344.085504:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830344.085507:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518247552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830344.085511:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518247680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830344.085524:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830344.085530:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830344/real 1787830344] req@ffff8b2e880761c0 x1874672518247680/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830344.085537:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518247680/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830344.085542:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830344.085544:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830344.085545:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518247680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830344.085548:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518247808:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830344.085562:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830344.085571:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830344/real 1787830344] req@ffff8b2e88074000 x1874672518247808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830344.085581:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518247808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830344.085595:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830344.085597:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830344.085601:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518247808:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830344.085606:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518247936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830344.085623:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830344.085643:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830344/real 1787830344] req@ffff8b2e88074b40 x1874672518247936/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830344.085676:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518247936/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830344.085683:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830344.085686:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830344.085689:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518247936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830344.085694:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518248064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830344.085712:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830344.085736:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830344/real 1787830344] req@ffff8b2e88075e00 x1874672518248064/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830344.085745:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518248064/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830369 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830344.085752:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830344.085755:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830344.085758:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518248064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830344.404811:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830344.917642:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830344.917769:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830345.108821:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830345.108898:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830345.428803:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830345.940814:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830345.940852:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830346.132792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830346.132792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830346.452804:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830346.964776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830346.964827:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830347.156815:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830347.157558:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830347.477525:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830347.988900:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830347.990091:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830348.181048:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830348.181164:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830348.500890:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830349.012801:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830349.012846:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830349.204803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830349.205149:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830349.205211:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830349.205217:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830349.205220:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830349.205229:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3355 00000100:00080000:3.0:1787830349.205240:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830349.205247:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830349.205335:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518248192/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830349.205407:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518248192:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830349.205463:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830349.205494:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830349.205498:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830349.205500:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830349.205504:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3355 00000100:00080000:3.0:1787830349.205508:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830349.205511:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830349.205520:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830349.205532:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518248320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830349.208201:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830349/real 1787830349] req@ffff8b2e880383c0 x1874672518248192/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830349.208227:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518248192/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830349.208237:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830349.208242:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830349.208245:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518248192:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830349.208273:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518248320:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830349.208319:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830349.208331:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830349/real 1787830349] req@ffff8b2e88038f00 x1874672518248320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830349.208339:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518248320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830349.208344:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830349.208348:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830349.208350:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518248320:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830349.208378:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830349.208383:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830349.208386:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830349.208392:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3355 00000100:00080000:3.0:1787830349.208398:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830349.208403:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830349.208414:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830349.208437:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518248448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830349.208466:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830349.208469:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830349.208470:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830349.208473:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3355 00000100:00080000:3.0:1787830349.208476:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830349.208478:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830349.208482:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830349.208490:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518248576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830349.208499:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830349.208503:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830349.208504:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830349.208508:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3355 00000100:00080000:3.0:1787830349.208511:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830349.208515:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830349.208520:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830349.208529:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518248704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830349.208538:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830349.208540:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830349.208542:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830349.208545:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3355 00000100:00080000:3.0:1787830349.208549:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830349.208552:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830349.208556:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830349.208566:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518248832/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830349.208576:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830349.208578:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830349.208580:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830349.208583:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3355 00000100:00080000:3.0:1787830349.208586:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830349.208589:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830349.208593:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830349.208603:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518248960/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830349.208668:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518248448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830349.208706:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830349.208720:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830349/real 1787830349] req@ffff8b2e88039a40 x1874672518248448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830349.208732:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518248448/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830349.208740:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830349.208743:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830349.208746:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518248448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830349.208752:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518248576:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830349.208771:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830349.208783:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830349/real 1787830349] req@ffff8b2e8803b0c0 x1874672518248576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830349.208794:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518248576/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830349.208800:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830349.208803:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830349.208806:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518248576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830349.208811:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518248704:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830349.208829:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830349.208840:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830349/real 1787830349] req@ffff8b2e88038f00 x1874672518248704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830349.208853:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518248704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830349.208861:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830349.208864:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830349.208868:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518248704:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830349.208873:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518248832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830349.208891:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830349.208901:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830349/real 1787830349] req@ffff8b2e880383c0 x1874672518248832/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830349.208911:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518248832/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830349.208918:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830349.208921:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830349.208923:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518248832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830349.208928:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518248960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830349.208944:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830349.208955:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830349/real 1787830349] req@ffff8b2e8803a580 x1874672518248960/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830349.208972:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518248960/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830374 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830349.208979:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830349.208981:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830349.208984:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518248960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830349.524770:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830350.038520:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830350.041781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830350.230095:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830350.230130:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830350.548791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830351.060762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830351.060849:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830351.252831:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830351.252832:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830351.572791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830352.085064:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830352.085335:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830352.277043:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830352.277489:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830352.598693:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830353.108793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830353.110977:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830353.300791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830353.300794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830353.620809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830354.132744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830354.132804:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830354.324787:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830354.324798:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830354.324803:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830354.324814:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3360 00000100:00080000:3.0:1787830354.324835:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830354.324843:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830354.324881:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518249088/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830354.324906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830354.324910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830354.324912:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830354.324924:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3360 00000100:00080000:3.0:1787830354.324928:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830354.324932:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830354.324942:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830354.324954:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518249216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830354.324963:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830354.324966:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830354.324968:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830354.324971:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3360 00000100:00080000:3.0:1787830354.324974:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830354.324978:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830354.324982:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830354.324991:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518249344/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830354.324999:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830354.325002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830354.325004:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830354.325007:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3360 00000100:00080000:3.0:1787830354.325010:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830354.325013:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830354.325017:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830354.325025:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518249472/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830354.325037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830354.325041:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830354.325043:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830354.325047:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3360 00000100:00080000:3.0:1787830354.325049:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830354.325052:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830354.325056:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830354.325064:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518249600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830354.325072:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830354.325075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830354.325076:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830354.325080:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3360 00000100:00080000:3.0:1787830354.325082:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830354.325085:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830354.325089:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830354.325096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518249728/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830354.325104:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830354.325107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830354.325109:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830354.325112:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3360 00000100:00080000:3.0:1787830354.325114:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830354.325117:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830354.325121:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830354.325131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518249856/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830354.325409:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830354.325528:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518249088:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830354.325585:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830354.325628:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830354/real 1787830354] req@ffff8b2e88039e00 x1874672518249088/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830354.325678:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518249088/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830354.325688:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830354.325692:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830354.325696:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518249088:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830354.325705:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518249216:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830354.325727:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830354.325739:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830354/real 1787830354] req@ffff8b2e8803b840 x1874672518249216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830354.325752:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518249216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830354.325759:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830354.325762:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830354.325765:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518249216:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830354.325771:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518249344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830354.325790:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830354.325800:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830354/real 1787830354] req@ffff8b2e88039680 x1874672518249344/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830354.325811:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518249344/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830354.325818:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830354.325821:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830354.325824:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518249344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830354.325829:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518249472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830354.325847:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830354.325858:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830354/real 1787830354] req@ffff8b2e880392c0 x1874672518249472/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830354.325869:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518249472/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830354.325877:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830354.325880:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830354.325883:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518249472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830354.325888:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518249600:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830354.325906:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830354.325917:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830354/real 1787830354] req@ffff8b2e8803a1c0 x1874672518249600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830354.325927:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518249600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830354.325934:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830354.325937:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830354.325940:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518249600:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830354.325945:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518249728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830354.325960:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830354.325969:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830354/real 1787830354] req@ffff8b2e8803b480 x1874672518249728/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830354.325979:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518249728/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830354.325991:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830354.325994:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830354.325997:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518249728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830354.326003:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518249856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830354.326021:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830354.326031:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830354/real 1787830354] req@ffff8b2e8803a580 x1874672518249856/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830354.326041:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518249856/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830379 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830354.326048:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830354.326052:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830354.326054:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518249856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830354.326146:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830354.644805:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830355.156811:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830355.156838:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830355.348790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830355.349996:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830355.668771:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830356.180787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830356.180913:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830356.372909:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830356.372913:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830356.692801:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830357.204782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830357.204858:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830357.396780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830357.396789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830357.716799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830358.228792:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830358.229030:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830358.420785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830358.421065:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830358.741234:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830359.252926:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830359.253147:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:3.0:1787830359.444774:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830359.444776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830359.444776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830359.444785:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830359.444789:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830359.444799:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3365 00000100:00080000:3.0:1787830359.444810:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830359.444817:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830359.444855:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518249984/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830359.444893:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830359.444897:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830359.444898:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830359.444902:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3365 00000100:00080000:3.0:1787830359.444906:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830359.444910:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830359.444919:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830359.444932:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518250112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830359.444941:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830359.444943:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830359.444944:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830359.444947:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3365 00000100:00080000:3.0:1787830359.444950:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830359.444952:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830359.444957:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830359.444958:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518249984:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830359.444966:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518250240/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830359.444973:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830359.444976:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830359.444977:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830359.444979:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3365 00000100:00080000:3.0:1787830359.444981:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830359.444983:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830359.444985:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830359.444993:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518250368/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830359.444998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830359.445001:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830359.445002:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830359.445004:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3365 00000100:00080000:3.0:1787830359.445006:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830359.445009:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830359.445011:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:2.0:1787830359.445024:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830359.445024:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518250496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830359.445035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830359.445054:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830359.445056:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830359.445061:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3365 00000100:00080000:3.0:1787830359.445066:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830359.445070:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830359.445073:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830359/real 1787830359] req@ffff8b2e880383c0 x1874672518249984/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787830359.445076:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830359.445089:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518250624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830359.445097:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518249984/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830359.445097:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830359.445102:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830359.445106:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830359.445114:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3365 00000100:00080000:3.0:1787830359.445120:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830359.445121:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830359.445127:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787830359.445128:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:3.0:1787830359.445135:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830359.445136:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518249984:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830359.445148:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518250112:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830359.445151:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518250752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830359.445177:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830359.445192:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830359/real 1787830359] req@ffff8b2e88038f00 x1874672518250112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830359.445203:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518250112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830359.445211:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830359.445214:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830359.445216:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518250112:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830359.445237:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518250240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830359.445254:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830359.445264:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830359/real 1787830359] req@ffff8b2e8803b0c0 x1874672518250240/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830359.445274:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518250240/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830359.445280:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830359.445289:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830359.445292:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518250240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830359.445298:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518250368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830359.445313:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830359.445322:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830359/real 1787830359] req@ffff8b2e88039a40 x1874672518250368/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830359.445331:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518250368/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830359.445339:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830359.445342:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830359.445345:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518250368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830359.445350:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518250496:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830359.445367:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830359.445377:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830359/real 1787830359] req@ffff8b2e8803a580 x1874672518250496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830359.445388:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518250496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830359.445396:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830359.445399:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830359.445402:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518250496:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830359.445408:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518250624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830359.445426:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830359.445437:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830359/real 1787830359] req@ffff8b2e8803b480 x1874672518250624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830359.445449:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518250624/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830359.445456:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830359.445459:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830359.445462:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518250624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830359.445467:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518250752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830359.445489:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830359.445500:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830359/real 1787830359] req@ffff8b2e8803a1c0 x1874672518250752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830359.445510:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518250752/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830384 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830359.445517:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830359.445521:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830359.445531:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518250752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830359.764812:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830360.276770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830360.277492:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830360.468772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830360.469408:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830360.788951:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830361.300768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830361.300800:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830361.492760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830361.492794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830361.812751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830362.324762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830362.324840:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830362.516777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830362.516840:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830362.836781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830363.348840:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830363.348878:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830363.540800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830363.540868:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830363.860807:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830364.373014:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830364.373090:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830364.564799:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830364.564834:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830364.564872:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830364.564879:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830364.564882:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830364.564890:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3370 00000100:00080000:1.0:1787830364.564900:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830364.564907:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830364.564941:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518250880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830364.564967:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830364.564970:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830364.564972:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830364.564976:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3370 00000100:00080000:1.0:1787830364.564980:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830364.564985:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830364.564994:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830364.565006:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518250880:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830364.565010:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518251008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830364.565022:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830364.565025:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830364.565027:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830364.565031:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3370 00000100:00080000:1.0:1787830364.565035:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830364.565038:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830364.565043:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830364.565055:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518251136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830364.565064:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830364.565069:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000800:00000100:2.0:1787830364.565070:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830364.565072:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830364.565078:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3370 00000100:00080000:1.0:1787830364.565082:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830364.565085:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830364.565089:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830364.565100:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518251264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830364.565108:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830364.565110:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830364.565112:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830364.565115:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3370 00000100:00080000:1.0:1787830364.565119:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830364.565122:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830364.565126:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830364.565135:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518251392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830364.565144:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830364.565146:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830364.565148:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830364.565151:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3370 00000100:00080000:1.0:1787830364.565154:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830364.565242:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830364/real 1787830364] req@ffff8b2e88076580 x1874672518250880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830364.565255:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518250880/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830364.565262:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830364.565267:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830364.565273:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518250880:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830364.565274:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830364.565293:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518251008:192.168.203.101@tcp:38:kworker.0 00000008:00100000:1.0:1787830364.565298:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830364.565314:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518251520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830364.565324:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830364.565329:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830364.565340:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830364/real 1787830364] req@ffff8b2e88075e00 x1874672518251008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830364.565351:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518251008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830364.565356:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830364.565358:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830364.565360:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518251008:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830364.565364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518251136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830364.565378:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830364.565384:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830364/real 1787830364] req@ffff8b2e88074b40 x1874672518251136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830364.565395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518251136/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830364.565400:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830364.565402:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830364.565404:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518251136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830364.565407:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518251264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830364.565415:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830364.565420:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830364/real 1787830364] req@ffff8b2e88074000 x1874672518251264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830364.565426:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518251264/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830364.565430:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830364.565432:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830364.565434:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518251264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830364.565436:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518251392:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830364.565444:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830364.565449:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830364/real 1787830364] req@ffff8b2e880761c0 x1874672518251392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830364.565455:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518251392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830364.565463:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830364.565465:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830364.565467:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518251392:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830364.565477:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518251520:192.168.203.101@tcp:8:kworker.0 00000100:00080000:1.0:1787830364.565485:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830364.565488:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:2.0:1787830364.565492:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830364.565497:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3370 00000100:00080000:1.0:1787830364.565503:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830364.565507:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830364/real 1787830364] req@ffff8b2e880770c0 x1874672518251520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830364.565510:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830364.565519:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830364.565527:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518251520/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830364.565535:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518251648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830364.565537:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830364.565545:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830364.565553:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518251520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830364.565567:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518251648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830364.565601:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830364.565613:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830364/real 1787830364] req@ffff8b2e88077840 x1874672518251648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830364.565624:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518251648/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830389 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830364.565631:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830364.565634:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830364.565637:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518251648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830364.885010:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830365.396993:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830365.397027:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830365.588794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830365.589228:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830365.908807:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830366.420757:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830366.420913:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830366.612793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830366.612793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830366.932810:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830367.444817:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830367.445571:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830367.636753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830367.637631:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830367.956780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830368.468780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830368.470187:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830368.660778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830368.660778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830368.980785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830369.492778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830369.492826:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830369.684814:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830369.684909:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830369.684915:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830369.684919:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830369.684928:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3376 00000100:00080000:0.0:1787830369.684938:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830369.684945:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830369.684983:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518251776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830369.685019:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830369.685023:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830369.685025:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830369.685029:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3376 00000100:00080000:0.0:1787830369.685033:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830369.685037:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830369.685047:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830369.685062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518251904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830369.685071:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830369.685074:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830369.685076:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830369.685079:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3376 00000100:00080000:0.0:1787830369.685082:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830369.685085:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830369.685090:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830369.685100:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518252032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830369.685109:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830369.685112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830369.685114:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830369.685117:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3376 00000100:00080000:0.0:1787830369.685121:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830369.685125:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830369.685130:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830369.685143:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518252160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830369.685143:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518251776:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787830369.685159:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830369.685161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830369.685175:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830369.685180:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3376 00000100:00080000:0.0:1787830369.685183:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830369.685187:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830369.685191:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830369.685202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518252288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830369.685213:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:2.0:1787830369.685213:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830369.685219:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830369.685221:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830369.685226:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3376 00000100:00080000:0.0:1787830369.685230:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830369.685234:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830369.685239:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830369.685250:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518252416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830369.685264:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830369/real 1787830369] req@ffff8b2e880403c0 x1874672518251776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830369.685266:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830369.685282:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830369.685284:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830369.685289:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518251776/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830369.685291:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3376 00000100:00080000:0.0:1787830369.685298:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830369.685304:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:2.0:1787830369.685307:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787830369.685310:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787830369.685314:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830369.685320:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518251776:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830369.685326:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518252544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830369.685332:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518251904:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830369.685362:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830369.685375:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830369/real 1787830369] req@ffff8b2e88040000 x1874672518251904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830369.685387:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518251904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830369.685395:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830369.685399:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830369.685402:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518251904:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830369.685408:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518252032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830369.685430:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830369.685441:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830369/real 1787830369] req@ffff8b2e880430c0 x1874672518252032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830369.685450:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518252032/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830369.685457:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830369.685460:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830369.685463:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518252032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830369.685478:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830369.685490:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518252160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830369.685518:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830369.685529:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830369/real 1787830369] req@ffff8b2e88043480 x1874672518252160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830369.685551:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518252160/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830369.685559:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830369.685562:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830369.685564:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518252160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830369.685570:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518252288:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830369.685588:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830369.685605:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830369/real 1787830369] req@ffff8b2e88042940 x1874672518252288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830369.685617:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518252288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830369.685634:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830369.685637:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830369.685640:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518252288:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830369.685667:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518252416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830369.685688:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830369.685699:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830369/real 1787830369] req@ffff8b2e88040f00 x1874672518252416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830369.685710:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518252416/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830369.685718:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830369.685721:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830369.685724:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518252416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830369.685728:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518252544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830369.685741:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830369.685748:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830369/real 1787830369] req@ffff8b2e88041a40 x1874672518252544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830369.685759:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518252544/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830394 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830369.685764:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830369.685766:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830369.685767:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518252544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830370.004806:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830370.516805:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830370.516890:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830370.708848:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830370.708848:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830371.028814:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830371.540800:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830371.540908:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830371.732795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830371.733711:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830372.052789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830372.564908:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830372.565152:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830372.756784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830372.756860:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830373.076802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830373.588776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830373.588822:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830373.780784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830373.780784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830374.101676:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830374.612789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830374.612895:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830374.804807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830374.804807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830374.804815:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830374.804821:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830374.804825:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830374.804836:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3381 00000100:00080000:0.0:1787830374.804848:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830374.804855:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830374.804941:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518252672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830374.804972:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830374.804975:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830374.804977:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830374.804981:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3381 00000100:00080000:0.0:1787830374.804985:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830374.804989:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830374.804998:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830374.805013:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518252800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830374.805021:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830374.805024:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830374.805026:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830374.805029:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3381 00000100:00080000:0.0:1787830374.805032:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830374.805035:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830374.805039:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830374.805049:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518252928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830374.805057:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830374.805060:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830374.805062:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830374.805065:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3381 00000100:00080000:0.0:1787830374.805068:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830374.805071:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830374.805075:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830374.805085:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518253056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830374.805097:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830374.805100:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830374.805101:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830374.805105:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3381 00000100:00080000:0.0:1787830374.805108:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830374.805112:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830374.805116:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830374.805125:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518253184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830374.805133:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830374.805136:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830374.805137:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830374.805140:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3381 00000100:00080000:0.0:1787830374.805143:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830374.805146:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830374.805150:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830374.805157:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518253312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830374.805165:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830374.805168:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830374.805169:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830374.805172:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3381 00000100:00080000:0.0:1787830374.805175:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830374.805178:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830374.805181:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830374.805188:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518253440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830374.805331:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518252672:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830374.805389:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830374.805433:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830374/real 1787830374] req@ffff8b2e880421c0 x1874672518252672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830374.805448:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518252672/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830374.805458:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830374.805462:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830374.805466:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518252672:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830374.805473:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518252800:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830374.805494:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830374.805522:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830374/real 1787830374] req@ffff8b2e88040b40 x1874672518252800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830374.805535:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518252800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830374.805542:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830374.805545:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830374.805547:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518252800:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830374.805553:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518252928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830374.805574:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830374.805585:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830374/real 1787830374] req@ffff8b2e880412c0 x1874672518252928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830374.805597:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518252928/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830374.805605:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830374.805608:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830374.805610:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518252928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830374.805617:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518253056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830374.805635:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830374.805671:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830374/real 1787830374] req@ffff8b2e88043840 x1874672518253056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830374.805684:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518253056/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830374.805696:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830374.805699:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830374.805702:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518253056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830374.805708:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518253184:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830374.805729:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830374.805741:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830374/real 1787830374] req@ffff8b2e88040780 x1874672518253184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830374.805752:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518253184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830374.805760:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830374.805763:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830374.805766:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518253184:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830374.805771:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518253312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830374.805790:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830374.805801:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830374/real 1787830374] req@ffff8b2e88042d00 x1874672518253312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830374.805812:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518253312/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830374.805819:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830374.805822:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830374.805826:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518253312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830374.805831:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518253440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830374.805850:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830374.805861:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830374/real 1787830374] req@ffff8b2e88042580 x1874672518253440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830374.805872:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518253440/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830399 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830374.805880:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830374.805882:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830374.805885:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518253440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830375.125229:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830375.636775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830375.637211:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830375.828787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830375.828936:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830376.149472:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830376.660883:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830376.661678:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830376.852793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830376.852906:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830377.172751:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830377.684775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830377.684967:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830377.876786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830377.876786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830378.197086:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830378.708809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830378.708833:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830378.900788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830378.900811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830379.220778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830379.732768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830379.733242:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830379.924796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830379.924859:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830379.924930:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830379.924937:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830379.924940:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830379.924951:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3386 00000100:00080000:1.0:1787830379.924964:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830379.924973:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830379.925032:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518253568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830379.925072:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830379.925076:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830379.925078:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830379.925083:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3386 00000100:00080000:1.0:1787830379.925087:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830379.925091:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830379.925102:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830379.925114:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518253696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830379.925123:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830379.925127:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830379.925128:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830379.925132:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3386 00000100:00080000:1.0:1787830379.925135:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830379.925139:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830379.925143:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830379.925154:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518253824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830379.925163:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830379.925166:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830379.925168:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830379.925172:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3386 00000100:00080000:1.0:1787830379.925175:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830379.925179:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830379.925183:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830379.925194:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518253952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830379.925202:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830379.925205:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830379.925207:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830379.925211:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3386 00000100:00080000:1.0:1787830379.925214:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830379.925217:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830379.925222:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830379.925233:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518254080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830379.925242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830379.925245:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830379.925247:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830379.925251:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3386 00000100:00080000:1.0:1787830379.925255:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830379.925259:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830379.925263:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830379.925273:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518254208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830379.925281:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830379.925284:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830379.925286:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830379.925289:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3386 00000100:00080000:1.0:1787830379.925292:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830379.925295:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830379.925299:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830379.925309:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518254336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830379.925398:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518253568:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830379.925484:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830379.925530:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830379/real 1787830379] req@ffff8b2e88077480 x1874672518253568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830379.925547:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518253568/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830379.925557:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830379.925560:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830379.925565:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518253568:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830379.925571:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518253696:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830379.925593:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830379.925606:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830379/real 1787830379] req@ffff8b2e88075680 x1874672518253696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830379.925618:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518253696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830379.925626:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830379.925629:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830379.925632:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518253696:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830379.925638:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518253824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830379.925688:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830379.925699:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830379/real 1787830379] req@ffff8b2e88074780 x1874672518253824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830379.925709:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518253824/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830379.925716:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830379.925719:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830379.925721:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518253824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830379.925726:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518253952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830379.925744:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830379.925755:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830379/real 1787830379] req@ffff8b2e88074f00 x1874672518253952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830379.925908:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518253952/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830379.925917:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830379.925920:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830379.925923:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518253952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830379.925928:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518254080:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830379.925950:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830379.925961:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830379/real 1787830379] req@ffff8b2e88075a40 x1874672518254080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830379.925972:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518254080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830379.925980:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830379.925983:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830379.925986:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518254080:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830379.925992:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518254208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830379.926012:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830379.926023:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830379/real 1787830379] req@ffff8b2e88076d00 x1874672518254208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830379.926033:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518254208/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830379.926041:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830379.926043:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830379.926046:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518254208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830379.926052:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518254336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830379.926071:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830379.926081:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830379/real 1787830379] req@ffff8b2e88077c00 x1874672518254336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830379.926091:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518254336/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830404 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830379.926099:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830379.926102:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830379.926105:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518254336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830380.244812:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830380.756775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830380.757839:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830380.948765:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830380.948875:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830381.268813:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830381.781163:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830381.781719:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830381.972813:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830381.972813:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830382.292805:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830382.804782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830382.805154:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830382.996783:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830382.998032:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830383.317238:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830383.828926:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830383.829055:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830384.020831:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830384.020941:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830384.340830:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830384.852836:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830384.853241:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830385.044789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830385.044792:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830385.044801:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830385.044804:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830385.044812:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3391 00000100:00080000:1.0:1787830385.044820:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830385.044826:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830385.044874:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518254464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830385.044909:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830385.044913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830385.044915:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830385.044920:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3391 00000100:00080000:1.0:1787830385.044926:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830385.044931:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830385.044942:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830385.044951:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830385.044958:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518254592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830385.044968:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830385.044972:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830385.044974:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830385.044977:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3391 00000100:00080000:1.0:1787830385.044980:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830385.044984:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830385.044988:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830385.044998:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518254720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830385.045004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830385.045006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830385.045007:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830385.045009:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3391 00000100:00080000:1.0:1787830385.045011:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830385.045013:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830385.045015:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830385.045020:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518254848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830385.045025:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830385.045027:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830385.045028:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830385.045030:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3391 00000100:00080000:1.0:1787830385.045031:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830385.045033:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830385.045035:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830385.045040:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518254976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830385.045045:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830385.045048:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830385.045048:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830385.045051:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3391 00000100:00080000:1.0:1787830385.045053:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830385.045058:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830385.045058:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518254464:192.168.203.101@tcp:250:kworker.0 00000008:00100000:1.0:1787830385.045064:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830385.045071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518255104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830385.045077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830385.045079:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830385.045079:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830385.045082:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3391 00000100:00080000:1.0:1787830385.045084:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830385.045086:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830385.045089:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830385.045096:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518255232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830385.045111:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830385.045263:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830385/real 1787830385] req@ffff8b2e88076940 x1874672518254464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830385.045281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518254464/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830385.045290:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830385.045294:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830385.045297:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518254464:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830385.045304:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518254592:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830385.045332:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830385.045345:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830385/real 1787830385] req@ffff8b2e88077c00 x1874672518254592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830385.045370:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518254592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830385.045379:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830385.045383:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830385.045386:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518254592:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830385.045395:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518254720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830385.045417:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830385.045437:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830385/real 1787830385] req@ffff8b2e88076d00 x1874672518254720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830385.045450:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518254720/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830385.045462:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830385.045466:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830385.045469:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518254720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830385.045474:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518254848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830385.045492:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830385.045502:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830385/real 1787830385] req@ffff8b2e88075a40 x1874672518254848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830385.045513:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518254848/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830385.045521:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830385.045523:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830385.045526:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518254848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830385.045532:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518254976:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830385.045546:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830385.045554:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830385/real 1787830385] req@ffff8b2e88074f00 x1874672518254976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830385.045565:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518254976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830385.045573:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830385.045575:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830385.045579:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518254976:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830385.045604:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518255104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830385.045620:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830385.045628:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830385/real 1787830385] req@ffff8b2e88074780 x1874672518255104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830385.045639:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518255104/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830385.045667:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830385.045670:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830385.045673:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518255104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830385.045678:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518255232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830385.045694:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830385.045703:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830385/real 1787830385] req@ffff8b2e88075680 x1874672518255232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830385.045713:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518255232/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830410 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830385.045720:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830385.045722:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830385.045725:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518255232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830385.364797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830385.876812:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830385.877515:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830386.068816:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830386.070819:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830386.388895:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830386.900778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830386.901350:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830387.093019:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830387.093091:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830387.412832:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830387.924996:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830387.925587:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830388.116809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830388.116884:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830388.436781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830388.948785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830388.949164:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830389.140801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830389.140801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830389.460969:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830389.972783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830389.972946:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830390.164789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830390.164872:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787830390.164911:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830390.164917:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830390.164921:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830390.164930:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3396 00000100:00080000:2.0:1787830390.164940:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830390.164946:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830390.164985:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518255360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830390.165012:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830390.165014:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830390.165016:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830390.165020:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3396 00000100:00080000:2.0:1787830390.165023:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830390.165026:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830390.165035:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830390.165047:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518255488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830390.165055:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830390.165058:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830390.165060:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830390.165063:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3396 00000100:00080000:2.0:1787830390.165067:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830390.165070:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830390.165074:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830390.165082:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518255616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830390.165095:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830390.165097:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830390.165098:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830390.165101:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3396 00000100:00080000:2.0:1787830390.165104:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830390.165107:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830390.165111:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830390.165120:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518255744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830390.165128:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830390.165131:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830390.165132:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830390.165136:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3396 00000100:00080000:2.0:1787830390.165138:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830390.165141:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830390.165145:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830390.165153:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518255872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830390.165172:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830390.165175:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830390.165177:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830390.165181:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3396 00000100:00080000:2.0:1787830390.165184:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830390.165187:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830390.165192:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830390.165201:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518256000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830390.165209:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830390.165212:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830390.165213:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830390.165217:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3396 00000100:00080000:2.0:1787830390.165219:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830390.165222:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830390.165226:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830390.165235:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518256128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830390.165568:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518255360:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830390.165681:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518255488:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830390.165712:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518255616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830390.165732:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518255744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830390.165759:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518255872:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830390.165780:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518256000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830390.165799:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518256128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830390.484731:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830390.997056:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830390.997706:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830391.188776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830391.188787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830391.508849:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830392.020771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830392.020804:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830392.213253:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830392.214182:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830392.532809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830393.044811:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830393.044889:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830393.236766:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:1.0:1787830393.236973:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787830393.237052:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787830393.237068:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830393.237118:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830393.237131:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830393.237143:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787830393.237143:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830390/real 1787830393] req@ffff8b2e8801de00 x1874672518255360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787830393.237156:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830393.237166:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787830393.237174:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518255360/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787830393.237178:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00080000:3.0:1787830393.237191:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830393.237197:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830393.237201:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518255360:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830393.237211:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830390/real 1787830393] req@ffff8b2e8801cb40 x1874672518255488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830393.237224:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518255488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830393.237231:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830393.237235:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830393.237238:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518255488:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830393.237245:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830390/real 1787830393] req@ffff8b2e8801c3c0 x1874672518255616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830393.237257:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518255616/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830393.237264:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830393.237267:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830393.237269:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518255616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830393.237273:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830390/real 1787830393] req@ffff8b2e8801d680 x1874672518255744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830393.237280:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518255744/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830393.237290:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830393.237292:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830393.237294:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518255744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830393.237298:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830390/real 1787830393] req@ffff8b2e8801f0c0 x1874672518255872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830393.237305:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518255872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830393.237309:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830393.237311:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830393.237313:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518255872:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830393.237316:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830390/real 1787830393] req@ffff8b2e8801c780 x1874672518256000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830393.237322:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518256000/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830393.237326:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830393.237328:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830393.237330:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518256000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830393.237333:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830390/real 1787830393] req@ffff8b2e8801ed00 x1874672518256128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830393.237339:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518256128/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830415 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830393.237342:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830393.237344:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830393.237346:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518256128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830393.237388:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830393.557091:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830394.068812:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830394.069864:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830394.260862:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830394.260862:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830394.581001:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830395.092811:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830395.092844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830395.284745:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830395.284755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830395.284908:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830395.284914:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830395.284918:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830395.284926:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3401 00000100:00080000:1.0:1787830395.284936:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830395.284942:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830395.284974:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518256256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830395.285007:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830395.285010:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830395.285012:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830395.285016:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3401 00000100:00080000:1.0:1787830395.285020:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830395.285023:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830395.285032:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830395.285045:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518256384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830395.285053:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830395.285056:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830395.285057:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830395.285060:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3401 00000100:00080000:1.0:1787830395.285063:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830395.285066:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830395.285070:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830395.285091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518256512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830395.285099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830395.285102:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830395.285103:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830395.285106:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3401 00000100:00080000:1.0:1787830395.285109:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830395.285113:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830395.285117:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830395.285127:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518256640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830395.285135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830395.285138:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830395.285139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830395.285143:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3401 00000100:00080000:1.0:1787830395.285146:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830395.285150:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830395.285154:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830395.285163:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518256768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830395.285171:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830395.285174:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830395.285176:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830395.285179:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3401 00000100:00080000:1.0:1787830395.285182:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830395.285185:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830395.285189:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830395.285198:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518256896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830395.285205:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830395.285208:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830395.285209:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830395.285212:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3401 00000100:00080000:1.0:1787830395.285216:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830395.285219:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830395.285223:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830395.285232:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518257024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830395.285305:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518256256:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830395.285374:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830395.285413:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830395/real 1787830395] req@ffff8b2e88077480 x1874672518256256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830395.285428:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518256256/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830395.285438:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830395.285442:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830395.285446:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518256256:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830395.285453:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518256384:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830395.285474:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830395.285486:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830395/real 1787830395] req@ffff8b2e88077840 x1874672518256384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830395.285498:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518256384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830395.285512:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830395.285515:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830395.285519:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518256384:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830395.285527:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518256512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830395.285546:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830395.285556:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830395/real 1787830395] req@ffff8b2e880770c0 x1874672518256512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830395.285566:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518256512/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830395.285573:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830395.285576:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830395.285578:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518256512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830395.285583:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518256640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830395.285600:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830395.285610:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830395/real 1787830395] req@ffff8b2e880761c0 x1874672518256640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830395.285621:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518256640/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830395.285628:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830395.285631:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830395.285634:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518256640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830395.285639:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518256768:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830395.285679:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830395.285691:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830395/real 1787830395] req@ffff8b2e88074000 x1874672518256768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830395.285703:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518256768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830395.285710:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830395.285714:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830395.285716:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518256768:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830395.285722:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518256896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830395.285740:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830395.285752:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830395/real 1787830395] req@ffff8b2e88074b40 x1874672518256896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830395.285762:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518256896/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830395.285770:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830395.285773:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830395.285776:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518256896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830395.285782:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518257024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830395.285800:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830395.285811:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830395/real 1787830395] req@ffff8b2e88075e00 x1874672518257024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830395.285822:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518257024/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830420 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830395.285836:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830395.285839:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830395.285842:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518257024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830395.604727:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830396.116878:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830396.117308:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830396.308792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830396.308971:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830396.628875:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830397.140971:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830397.141242:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830397.332830:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830397.332937:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830397.652786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830398.164813:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830398.164895:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830398.357463:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830398.357463:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830398.678546:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830399.188771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830399.188868:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830399.380811:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830399.380811:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830399.700786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830400.213606:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830400.214412:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830400.404879:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830400.405006:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830400.406311:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830400.406319:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830400.406331:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830400.406341:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3406 00000100:00080000:1.0:1787830400.406352:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830400.406358:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830400.406395:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518257152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830400.406427:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830400.406431:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830400.406433:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830400.406437:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3406 00000100:00080000:1.0:1787830400.406440:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830400.406444:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830400.406453:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830400.406467:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518257280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830400.406475:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830400.406479:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830400.406481:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830400.406483:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3406 00000100:00080000:1.0:1787830400.406486:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830400.406489:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830400.406493:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830400.406502:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518257408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830400.406510:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830400.406513:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830400.406515:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830400.406518:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3406 00000100:00080000:1.0:1787830400.406520:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830400.406523:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830400.406527:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830400.406536:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518257536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830400.406543:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830400.406546:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830400.406548:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830400.406551:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3406 00000100:00080000:1.0:1787830400.406554:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830400.406557:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830400.406562:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830400.406571:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518257664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830400.406580:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830400.406583:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830400.406584:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830400.406588:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3406 00000100:00080000:1.0:1787830400.406590:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830400.406599:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830400.406603:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830400.406613:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518257792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830400.406620:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830400.406623:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830400.406625:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830400.406628:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3406 00000100:00080000:1.0:1787830400.406632:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830400.406635:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830400.406639:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830400.406666:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518257920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830400.406794:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518257152:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830400.406860:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830400.406904:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830400/real 1787830400] req@ffff8b2e88076580 x1874672518257152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830400.406920:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518257152/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830400.406929:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830400.406933:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830400.406937:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518257152:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830400.406944:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518257280:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830400.406966:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830400.406979:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830400/real 1787830400] req@ffff8b2e88075e00 x1874672518257280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830400.406996:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518257280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830400.407003:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830400.407006:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830400.407008:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518257280:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830400.407013:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518257408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830400.407030:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830400.407038:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830400/real 1787830400] req@ffff8b2e88074b40 x1874672518257408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830400.407047:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518257408/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830400.407053:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830400.407055:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830400.407058:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518257408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830400.407062:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518257536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830400.407073:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830400.407080:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830400/real 1787830400] req@ffff8b2e88074000 x1874672518257536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830400.407088:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518257536/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830400.407098:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830400.407101:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830400.407103:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518257536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830400.407107:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518257664:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830400.407117:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830400.407124:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830400/real 1787830400] req@ffff8b2e880761c0 x1874672518257664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830400.407132:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518257664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830400.407138:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830400.407140:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830400.407143:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518257664:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830400.407147:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518257792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830400.407157:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830400.407163:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830400/real 1787830400] req@ffff8b2e880770c0 x1874672518257792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830400.407171:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518257792/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830400.407177:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830400.407179:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830400.407182:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518257792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830400.407186:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518257920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830400.407197:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830400.407204:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830400/real 1787830400] req@ffff8b2e88077840 x1874672518257920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830400.407212:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518257920/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830425 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830400.407218:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830400.407220:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830400.407222:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518257920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830400.724791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830401.236792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830401.236856:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830401.428789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830401.429395:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830401.749163:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830402.260968:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830402.261722:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830402.452809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830402.452809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830402.772977:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830403.284816:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830403.287426:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830403.476781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830403.478408:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830403.796782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830404.308861:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830404.308944:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830404.501332:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830404.501560:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830404.820810:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830405.332765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830405.332854:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830405.524740:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830405.524741:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830405.524751:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830405.524755:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830405.524756:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830405.524764:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3411 00000100:00080000:1.0:1787830405.524776:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830405.524783:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830405.524818:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518258048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830405.524869:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518258048:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830405.524874:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830405.524879:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830405.524880:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830405.524885:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3411 00000100:00080000:1.0:1787830405.524890:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830405.524894:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830405.524903:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830405.524916:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518258176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830405.524925:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830405.524930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830405.524934:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830405.524936:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830405.524941:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3411 00000100:00080000:1.0:1787830405.524945:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830405.524948:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830405.524953:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830405.524962:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518258304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830405.524972:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830405/real 1787830405] req@ffff8b2e88077480 x1874672518258048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830405.524973:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830405.524980:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830405.524983:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830405.524990:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3411 00000100:00080000:1.0:1787830405.524995:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830405.524998:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830405.525002:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518258048/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787830405.525004:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787830405.525019:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787830405.525020:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518258432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830405.525027:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830405.525035:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518258048:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830405.525036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830405.525045:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830405.525047:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830405.525053:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3411 00000100:00080000:1.0:1787830405.525056:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830405.525060:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830405.525065:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830405.525068:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518258176:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830405.525078:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518258560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830405.525089:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830405.525092:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830405.525096:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:3.0:1787830405.525097:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830405.525103:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3411 00000100:00080000:1.0:1787830405.525108:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830405.525112:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830405.525117:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830405.525118:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830405/real 1787830405] req@ffff8b2e88075680 x1874672518258176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830405.525129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518258688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830405.525135:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518258176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830405.525144:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830405.525151:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830405.525152:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830405.525153:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830405.525162:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3411 00000100:00080000:3.0:1787830405.525163:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787830405.525165:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830405.525170:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518258176:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830405.525172:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830405.525180:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830405.525185:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518258304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830405.525191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518258816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830405.525212:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830405.525224:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830405/real 1787830405] req@ffff8b2e88074780 x1874672518258304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830405.525234:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518258304/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830405.525240:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830405.525243:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830405.525246:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518258304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830405.525251:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518258432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830405.525267:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830405.525277:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830405/real 1787830405] req@ffff8b2e88074f00 x1874672518258432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830405.525299:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518258432/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830405.525308:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830405.525311:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830405.525314:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518258432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830405.525328:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518258560:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830405.525346:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830405.525357:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830405/real 1787830405] req@ffff8b2e88075a40 x1874672518258560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830405.525367:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518258560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830405.525375:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830405.525378:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830405.525381:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518258560:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830405.525386:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518258688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830405.525401:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830405.525410:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830405/real 1787830405] req@ffff8b2e88076d00 x1874672518258688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830405.525421:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518258688/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830405.525428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830405.525431:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830405.525444:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518258688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830405.525450:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518258816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830405.525468:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830405.525478:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830405/real 1787830405] req@ffff8b2e88077c00 x1874672518258816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830405.525488:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518258816/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830430 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830405.525495:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830405.525498:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830405.525500:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518258816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830405.844778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830406.356824:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830406.357235:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830406.549057:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830406.549145:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830406.868820:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830407.380940:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830407.384697:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830407.572805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830407.573626:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830407.892783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830408.404774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830408.405815:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830408.596778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830408.596778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830408.916896:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830409.428892:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830409.429815:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830409.620786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830409.620786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830409.940990:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830410.452826:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830410.452862:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830410.644786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830410.644786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830410.644872:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830410.644878:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830410.644891:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830410.644900:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3417 00000100:00080000:3.0:1787830410.644910:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830410.644917:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830410.644952:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518258944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830410.644989:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830410.644993:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830410.644995:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830410.644999:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3417 00000100:00080000:3.0:1787830410.645003:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830410.645007:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830410.645016:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830410.645027:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518259072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830410.645037:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830410.645040:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830410.645043:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830410.645047:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3417 00000100:00080000:3.0:1787830410.645051:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830410.645060:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830410.645065:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830410.645068:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518258944:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830410.645079:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518259200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830410.645090:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830410.645094:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830410.645096:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830410.645100:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3417 00000100:00080000:3.0:1787830410.645103:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830410.645107:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830410.645111:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830410.645122:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518259328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830410.645129:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830410.645138:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830410.645141:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830410.645143:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830410.645148:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3417 00000100:00080000:3.0:1787830410.645152:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830410.645156:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830410.645160:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830410.645173:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518259456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830410.645185:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830410.645189:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830410.645191:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830410.645194:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3417 00000100:00080000:3.0:1787830410.645197:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830410.645200:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830410.645204:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830410.645213:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518259584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830410.645220:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830410.645223:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830410.645225:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830410.645228:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3417 00000100:00080000:3.0:1787830410.645230:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830410.645233:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830410.645238:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830410.645247:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518259712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830410.645263:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830410/real 1787830410] req@ffff8b2e880392c0 x1874672518258944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830410.645280:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518258944/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830410.645288:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830410.645291:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830410.645294:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518258944:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830410.645300:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518259072:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830410.645321:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830410.645332:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830410/real 1787830410] req@ffff8b2e88039680 x1874672518259072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830410.645342:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518259072/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830410.645350:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830410.645352:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830410.645355:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518259072:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830410.645376:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518259200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830410.645397:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830410.645411:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830410/real 1787830410] req@ffff8b2e8803b840 x1874672518259200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830410.645425:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518259200/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830410.645436:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830410.645439:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830410.645442:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518259200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830410.645447:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518259328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830410.645463:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830410.645472:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830410/real 1787830410] req@ffff8b2e88039e00 x1874672518259328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830410.645481:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518259328/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830410.645487:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830410.645490:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830410.645493:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518259328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830410.645498:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518259456:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830410.645516:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830410.645526:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830410/real 1787830410] req@ffff8b2e8803a1c0 x1874672518259456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830410.645537:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518259456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830410.645544:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830410.645547:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830410.645550:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518259456:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830410.645556:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518259584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830410.645573:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830410.645584:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830410/real 1787830410] req@ffff8b2e8803b480 x1874672518259584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830410.645594:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518259584/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830410.645601:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830410.645603:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830410.645606:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518259584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830410.645621:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518259712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830410.645638:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830410.645671:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830410/real 1787830410] req@ffff8b2e8803a580 x1874672518259712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830410.645696:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518259712/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830435 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830410.645702:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830410.645705:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830410.645707:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518259712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830410.964928:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830411.476809:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830411.476988:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830411.669307:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830411.669315:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830411.988948:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830412.500916:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830412.501535:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830412.692791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830412.692921:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830413.014685:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830413.524835:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830413.524891:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830413.717471:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830413.717771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830414.036775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830414.549218:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830414.549911:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830414.740788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830414.740788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830415.060845:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830415.572778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830415.572806:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830415.764945:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830415.765001:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830415.765007:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830415.765010:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830415.765020:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3422 00000100:00080000:1.0:1787830415.765031:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830415.765038:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830415.765087:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518259840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830415.765109:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830415.765111:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830415.765112:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830415.765115:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3422 00000100:00080000:1.0:1787830415.765118:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830415.765120:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830415.765128:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830415.765138:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518259968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830415.765148:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830415.765150:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830415.765152:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830415.765155:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3422 00000100:00080000:1.0:1787830415.765158:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830415.765161:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830415.765165:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830415.765174:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518260096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830415.765183:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830415.765186:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830415.765188:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830415.765191:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3422 00000100:00080000:1.0:1787830415.765194:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830415.765196:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830415.765199:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830415.765206:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518260224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830415.765211:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830415.765213:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830415.765214:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830415.765226:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3422 00000100:00080000:1.0:1787830415.765229:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830415.765233:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830415.765237:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830415.765246:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518260352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830415.765253:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830415.765256:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830415.765257:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830415.765261:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3422 00000100:00080000:1.0:1787830415.765264:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830415.765267:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830415.765271:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830415.765278:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518260480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830415.765286:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830415.765289:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830415.765290:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830415.765293:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3422 00000100:00080000:1.0:1787830415.765296:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830415.765300:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830415.765304:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830415.765312:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518260608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830415.765455:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518259840:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830415.765510:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830415.765551:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830415/real 1787830415] req@ffff8b2e88076940 x1874672518259840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830415.765566:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518259840/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830415.765575:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830415.765578:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830415.765582:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518259840:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830415.765588:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518259968:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830415.765609:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830415.765620:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830415/real 1787830415] req@ffff8b2e88077c00 x1874672518259968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830415.765631:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518259968/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830415.765638:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830415.765641:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830415.765665:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518259968:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830415.765672:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518260096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830415.765693:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830415.765703:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830415/real 1787830415] req@ffff8b2e88076d00 x1874672518260096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830415.765714:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518260096/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830415.765721:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830415.765725:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830415.765728:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518260096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830415.765733:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518260224:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830415.765750:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830415.765760:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830415/real 1787830415] req@ffff8b2e88075a40 x1874672518260224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830415.765770:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518260224/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830415.765778:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830415.765781:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830415.765784:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518260224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830415.765789:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518260352:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830415.765805:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830415.765920:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830415.766285:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830415/real 1787830415] req@ffff8b2e88074f00 x1874672518260352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830415.766309:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518260352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830415.766319:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830415.766322:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830415.766324:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518260352:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830415.766330:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518260480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830415.766354:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830415.766366:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830415/real 1787830415] req@ffff8b2e88074780 x1874672518260480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830415.766377:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518260480/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830415.766384:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830415.766387:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830415.766390:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518260480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830415.766395:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518260608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830415.766414:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830415.766424:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830415/real 1787830415] req@ffff8b2e88075680 x1874672518260608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830415.766435:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518260608/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830440 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830415.766448:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830415.766451:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830415.766454:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518260608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830416.085378:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830416.596875:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830416.596981:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830416.788761:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830416.789295:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830417.109180:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830417.621876:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830417.622119:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830417.812754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830417.813360:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830418.132762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830418.645168:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830418.645188:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830418.836779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830418.837247:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830419.156784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830419.668980:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830419.669141:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830419.860803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830419.860847:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830420.180850:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830420.692790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830420.692883:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830420.884775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830420.884789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830420.884893:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830420.884898:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830420.884902:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830420.884911:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3427 00000100:00080000:1.0:1787830420.884921:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830420.884928:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830420.884966:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518260736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830420.884996:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830420.885000:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830420.885002:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830420.885006:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3427 00000100:00080000:1.0:1787830420.885010:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830420.885014:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830420.885023:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830420.885035:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518260864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830420.885044:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830420.885047:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830420.885049:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830420.885052:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3427 00000100:00080000:1.0:1787830420.885055:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830420.885058:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830420.885063:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830420.885072:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518260992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830420.885079:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830420.885082:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830420.885084:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830420.885086:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3427 00000100:00080000:1.0:1787830420.885089:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830420.885092:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830420.885096:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830420.885104:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518261120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830420.885111:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830420.885114:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830420.885115:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830420.885119:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3427 00000100:00080000:1.0:1787830420.885121:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830420.885125:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830420.885128:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830420.885136:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518261248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830420.885144:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830420.885147:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830420.885149:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830420.885152:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3427 00000100:00080000:1.0:1787830420.885155:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830420.885157:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830420.885161:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830420.885170:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518261376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830420.885188:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830420.885191:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830420.885193:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830420.885197:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3427 00000100:00080000:1.0:1787830420.885200:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830420.885203:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830420.885207:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830420.885215:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518261504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830420.885279:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518260736:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830420.885332:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830420.885385:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830420/real 1787830420] req@ffff8b2e88077480 x1874672518260736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830420.885399:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518260736/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830420.885407:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830420.885411:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830420.885414:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518260736:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830420.885421:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518260864:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830420.885439:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830420.885452:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830420/real 1787830420] req@ffff8b2e88077840 x1874672518260864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830420.885467:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518260864/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830420.885473:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830420.885475:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830420.885478:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518260864:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830420.885483:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518260992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830420.885499:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830420.885508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830420/real 1787830420] req@ffff8b2e880770c0 x1874672518260992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830420.885516:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518260992/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830420.885523:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830420.885525:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830420.885528:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518260992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830420.885532:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518261120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830420.885547:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830420.885557:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830420/real 1787830420] req@ffff8b2e880761c0 x1874672518261120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830420.885567:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518261120/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830420.885580:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830420.885583:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830420.885586:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518261120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830420.885592:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518261248:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830420.885609:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830420.885618:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830420/real 1787830420] req@ffff8b2e88074000 x1874672518261248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830420.885628:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518261248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830420.885636:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830420.885639:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830420.885641:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518261248:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830420.885665:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518261376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830420.885682:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830420.885691:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830420/real 1787830420] req@ffff8b2e88074b40 x1874672518261376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830420.885701:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518261376/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830420.885708:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830420.885710:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830420.885713:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518261376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830420.885718:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518261504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830420.885734:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830420.885742:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830420/real 1787830420] req@ffff8b2e88075e00 x1874672518261504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830420.885751:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518261504/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830445 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830420.885758:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830420.885761:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830420.885763:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518261504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830421.204801:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830421.716810:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830421.716950:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830421.908772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830421.908779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830422.228918:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830422.740807:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830422.740890:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830422.932762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830422.932893:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830423.252801:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830423.764796:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830423.764977:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830423.956769:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830423.957014:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830424.276795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830424.788803:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830424.788813:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830424.980788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830424.981833:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830425.300800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:0.0:1787830425.812807:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830425.812827:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:2.0:1787830426.004799:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830426.004811:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830426.004815:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830426.004825:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3432 00000100:00080000:2.0:1787830426.004836:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830426.004843:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830426.004855:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830426.004879:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518261632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830426.004906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830426.004910:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830426.004912:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830426.004917:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3432 00000100:00080000:2.0:1787830426.004921:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830426.004924:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830426.004935:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830426.004936:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830426.004953:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518261760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830426.004961:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830426.004964:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830426.004966:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830426.004969:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3432 00000100:00080000:2.0:1787830426.004972:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830426.004976:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830426.004980:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830426.004991:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518261888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830426.005001:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830426.005003:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830426.005005:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830426.005008:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3432 00000100:00080000:2.0:1787830426.005011:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830426.005015:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830426.005019:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830426.005020:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518261632:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830426.005030:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518262016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830426.005040:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830426.005043:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830426.005045:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830426.005049:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3432 00000100:00080000:2.0:1787830426.005062:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830426.005066:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830426.005071:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787830426.005077:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830426.005082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518262144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830426.005091:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830426.005095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830426.005097:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830426.005101:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3432 00000100:00080000:2.0:1787830426.005104:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830426.005108:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830426.005112:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830426.005121:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518262272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830426.005129:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830426.005132:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830426.005133:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830426.005136:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3432 00000100:00080000:2.0:1787830426.005139:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830426.005142:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830426.005157:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830426.005168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518262400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830426.005176:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830426/real 1787830426] req@ffff8b2e8801da40 x1874672518261632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830426.005197:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518261632/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830426.005207:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830426.005210:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830426.005214:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518261632:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830426.005221:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518261760:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830426.005244:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830426.005256:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830426/real 1787830426] req@ffff8b2e8801fc00 x1874672518261760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830426.005268:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518261760/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830426.005277:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830426.005279:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830426.005281:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518261760:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830426.005307:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518261888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830426.005328:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830426.005344:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830426/real 1787830426] req@ffff8b2e8801f480 x1874672518261888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830426.005357:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518261888/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830426.005363:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830426.005365:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830426.005367:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518261888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830426.005371:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518262016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830426.005384:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830426.005394:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830426/real 1787830426] req@ffff8b2e8801cf00 x1874672518262016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830426.005405:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518262016/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830426.005510:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830426.005513:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830426.005515:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518262016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830426.005520:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518262144:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830426.005543:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830426.005562:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830426/real 1787830426] req@ffff8b2e8801c000 x1874672518262144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830426.005573:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518262144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830426.005582:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830426.005585:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830426.005588:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518262144:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830426.005593:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518262272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830426.005612:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830426.005623:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830426/real 1787830426] req@ffff8b2e8801e580 x1874672518262272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830426.005669:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518262272/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830426.005677:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830426.005679:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830426.005682:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518262272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830426.005686:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518262400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830426.005707:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830426.005718:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830426/real 1787830426] req@ffff8b2e8801e940 x1874672518262400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830426.005730:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518262400/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830451 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830426.005737:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830426.005740:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830426.005742:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518262400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830426.324762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830426.836836:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830426.836908:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830427.028854:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830427.029012:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830427.348861:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830427.860789:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830427.860911:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830428.052778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830428.052778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830428.372749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830428.884769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830428.884883:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830429.076804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830429.076804:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830429.396862:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830429.908781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830429.910142:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830430.100771:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830430.100771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830430.420810:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830430.932868:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830430.933245:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:0.0:1787830431.124779:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830431.124779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830431.124790:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830431.124795:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830431.124804:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3437 00000100:00080000:0.0:1787830431.124816:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830431.124828:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830431.124846:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830431.124869:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518262528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830431.124898:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830431.124901:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830431.124903:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830431.124906:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3437 00000100:00080000:0.0:1787830431.124909:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830431.124911:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830431.124919:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830431.124931:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518262656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830431.124944:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830431.124947:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830431.124949:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830431.124952:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3437 00000100:00080000:0.0:1787830431.124955:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830431.124958:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830431.124963:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830431.124972:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518262784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830431.124981:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830431.124983:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830431.124985:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830431.124989:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3437 00000100:00080000:0.0:1787830431.124991:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830431.125000:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830431.125005:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830431.125014:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518262912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830431.125022:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830431.125024:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830431.125025:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830431.125030:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3437 00000100:00100000:3.0:1787830431.125032:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518262528:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787830431.125033:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830431.125038:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830431.125041:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830431.125049:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518263040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830431.125062:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830431.125064:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830431.125066:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830431.125070:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3437 00000100:00080000:0.0:1787830431.125073:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830431.125077:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830431.125081:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830431.125102:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518263168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830431.125122:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787830431.125123:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830431.125129:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830431.125131:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830431.125135:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3437 00000100:00080000:0.0:1787830431.125139:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830431.125142:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830431.125147:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830431.125158:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518263296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830431.125175:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830431/real 1787830431] req@ffff8b2e88041e00 x1874672518262528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830431.125193:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518262528/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830431.125203:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830431.125207:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830431.125212:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518262528:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830431.125218:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518262656:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830431.125239:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830431.125252:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830431/real 1787830431] req@ffff8b2e88042580 x1874672518262656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830431.125265:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518262656/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830431.125273:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830431.125276:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830431.125279:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518262656:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830431.125285:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518262784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830431.125304:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830431.125313:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830431/real 1787830431] req@ffff8b2e88042d00 x1874672518262784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830431.125322:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518262784/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830431.125339:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830431.125342:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830431.125345:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518262784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830431.125367:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518262912:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830431.125385:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830431.125397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830431/real 1787830431] req@ffff8b2e88040780 x1874672518262912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830431.125408:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518262912/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830431.125415:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830431.125417:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830431.125420:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518262912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830431.125426:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518263040:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830431.125455:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830431.125466:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830431/real 1787830431] req@ffff8b2e88043840 x1874672518263040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830431.125476:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518263040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830431.125484:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830431.125487:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830431.125490:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518263040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830431.125496:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518263168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830431.125511:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830431.125520:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830431/real 1787830431] req@ffff8b2e880412c0 x1874672518263168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830431.125530:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518263168/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830431.125538:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830431.125540:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830431.125543:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518263168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830431.125548:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518263296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830431.125564:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830431.125574:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830431/real 1787830431] req@ffff8b2e88040b40 x1874672518263296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830431.125584:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518263296/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830456 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830431.125592:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830431.125595:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830431.125598:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518263296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830431.444856:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830431.956788:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830431.956802:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830432.148833:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830432.149114:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830432.468955:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830432.980715:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830432.981383:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830433.172782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830433.172902:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830433.493180:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830434.004710:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830434.004820:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830434.196772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830434.196969:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830434.516939:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830435.028783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830435.028834:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830435.222183:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830435.222183:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830435.540795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830436.052921:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830436.052935:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830436.244782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830436.245762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830436.247215:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830436.247223:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830436.247227:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830436.247236:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3442 00000100:00080000:3.0:1787830436.247249:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830436.247257:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830436.247295:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518263424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830436.247359:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518263424:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830436.247414:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830436.247446:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830436.247450:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830436.247452:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830436.247457:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3442 00000100:00080000:3.0:1787830436.247460:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830436.247465:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830436.247474:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830436.247486:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518263552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830436.247957:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830436/real 1787830436] req@ffff8b2e88039a40 x1874672518263424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830436.247975:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518263424/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830436.247985:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830436.247990:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830436.247994:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518263424:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830436.248015:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518263552:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830436.248042:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830436.248056:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830436/real 1787830436] req@ffff8b2e8803b0c0 x1874672518263552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830436.248068:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518263552/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830436.248089:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830436.248092:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830436.248096:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518263552:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830436.248224:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830436.248229:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830436.248231:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830436.248235:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3442 00000100:00080000:3.0:1787830436.248240:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830436.248244:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830436.248251:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830436.248266:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518263680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830436.248311:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518263680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830436.248335:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830436.248359:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830436.248363:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830436.248365:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830436.248368:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3442 00000100:00080000:3.0:1787830436.248373:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830436.248376:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830436.248382:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830436.248393:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518263808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830436.248514:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830436/real 1787830436] req@ffff8b2e88038f00 x1874672518263680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830436.248529:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518263680/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830436.248537:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830436.248539:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830436.248542:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518263680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830436.248555:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518263808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830436.248576:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830436.248587:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830436/real 1787830436] req@ffff8b2e880383c0 x1874672518263808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830436.248597:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518263808/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830436.248605:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830436.248608:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830436.248611:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518263808:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830436.248756:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830436.248760:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830436.248762:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830436.248767:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3442 00000100:00080000:3.0:1787830436.248771:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830436.248775:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830436.248780:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830436.248793:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518263936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830436.248809:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830436.248813:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830436.248815:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830436.248822:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3442 00000100:00080000:3.0:1787830436.248826:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830436.248830:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830436.248836:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830436.248847:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518264064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830436.248851:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518263936:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830436.248865:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830436.248868:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830436.248871:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830436.248875:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3442 00000100:00080000:3.0:1787830436.248879:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787830436.248884:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830436.248885:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830436.248892:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830436.248899:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830436/real 1787830436] req@ffff8b2e880383c0 x1874672518263936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830436.248907:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518264192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830436.248916:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518263936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830436.248928:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830436.248932:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830436.248936:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518263936:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830436.248952:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518264064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830436.248971:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830436.248982:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830436/real 1787830436] req@ffff8b2e88038f00 x1874672518264064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830436.248994:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518264064/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830436.249001:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830436.249004:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830436.249007:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518264064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830436.249013:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518264192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830436.249031:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830436.249041:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830436/real 1787830436] req@ffff8b2e8803b0c0 x1874672518264192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830436.249051:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518264192/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830461 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830436.249058:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830436.249061:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830436.249064:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518264192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830436.564793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830437.077210:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830437.077251:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830437.268751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830437.268751:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830437.588788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830438.100819:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830438.101256:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830438.292775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830438.292779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830438.613113:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830439.124784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830439.124859:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830439.317038:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830439.318300:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830439.637192:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830440.148786:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830440.148900:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830440.340785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830440.340785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830440.661032:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830441.172775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:0.0:1787830441.174051:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830441.364773:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830441.364783:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00100000:3.0:1787830441.364784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787830441.364788:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830441.364799:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3447 00000100:00080000:2.0:1787830441.364808:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830441.364815:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830441.364853:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518264320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830441.364879:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830441.364883:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830441.364887:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830441.364891:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3447 00000100:00080000:2.0:1787830441.364894:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830441.364898:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830441.364907:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830441.364919:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518264448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830441.364929:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518264320:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787830441.364930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830441.364937:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830441.364940:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830441.364944:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3447 00000100:00080000:2.0:1787830441.364946:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830441.364949:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830441.364954:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830441.364962:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518264576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830441.364977:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830441.364979:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830441.364982:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830441.364985:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3447 00000100:00080000:2.0:1787830441.364988:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830441.364991:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787830441.364996:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:2.0:1787830441.364997:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830441.365008:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518264704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830441.365016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830441.365020:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830441.365021:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830441.365030:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3447 00000100:00080000:2.0:1787830441.365034:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830441.365049:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830441.365055:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830441/real 1787830441] req@ffff8b2e8801e940 x1874672518264320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787830441.365055:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830441.365069:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518264832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830441.365079:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518264320/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830441.365080:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830441.365086:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830441.365090:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830441.365098:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3447 00000100:00080000:0.0:1787830441.365100:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830441.365102:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830441.365107:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787830441.365108:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830441.365116:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518264320:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787830441.365124:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830441.365138:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518264960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830441.365147:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518264448:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787830441.365148:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830441.365155:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830441.365159:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830441.365164:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3447 00000100:00080000:2.0:1787830441.365167:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830441.365171:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830441.365176:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830441.365182:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830441.365189:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518265088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830441.365200:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830441/real 1787830441] req@ffff8b2e8801e580 x1874672518264448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830441.365211:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830441.365215:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518264448/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830441.365225:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830441.365229:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830441.365233:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518264448:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830441.365240:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518264576:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830441.365260:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830441.365271:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830441/real 1787830441] req@ffff8b2e8801c000 x1874672518264576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830441.365281:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518264576/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830441.365288:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830441.365291:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830441.365293:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518264576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830441.365297:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518264704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830441.365310:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830441.365320:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830441/real 1787830441] req@ffff8b2e8801cf00 x1874672518264704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830441.365337:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518264704/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830441.365345:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830441.365348:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830441.365351:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518264704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830441.365356:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518264832:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830441.365374:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830441.365385:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830441/real 1787830441] req@ffff8b2e8801f480 x1874672518264832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830441.365395:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518264832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830441.365404:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830441.365406:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830441.365409:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518264832:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830441.365426:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518264960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830441.365445:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830441.365456:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830441/real 1787830441] req@ffff8b2e8801fc00 x1874672518264960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830441.365466:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518264960/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830441.365480:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830441.365483:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830441.365486:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518264960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830441.365492:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518265088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830441.365511:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830441.365522:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830441/real 1787830441] req@ffff8b2e8801da40 x1874672518265088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830441.365532:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518265088/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830466 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830441.365540:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830441.365543:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830441.365546:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518265088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830441.684801:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830442.196778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830442.196913:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830442.388772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830442.390178:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830442.708918:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830443.220765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830443.220864:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830443.412835:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830443.412843:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830443.732961:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830444.244832:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830444.244882:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830444.437158:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830444.437369:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830444.756788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830445.268760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830445.268863:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830445.460770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830445.461052:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830445.780983:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830446.292762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830446.292944:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:2.0:1787830446.484798:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830446.484809:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830446.484814:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830446.484817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787830446.484826:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3452 00000100:00100000:1.0:1787830446.484833:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787830446.484838:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830446.484850:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830446.484887:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518265216/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830446.484911:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830446.484914:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830446.484916:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830446.484920:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3452 00000100:00080000:2.0:1787830446.484924:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830446.484928:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830446.484938:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830446.484948:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518265344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830446.484960:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830446.484963:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518265216:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787830446.484964:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830446.484967:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830446.484974:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3452 00000100:00080000:2.0:1787830446.484977:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830446.484981:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830446.484986:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830446.484997:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518265472/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830446.485053:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830446.485108:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830446/real 1787830446] req@ffff8b2e8801ed00 x1874672518265216/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830446.485131:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518265216/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830446.485139:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830446.485144:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830446.485145:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830446.485147:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830446.485152:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3452 00000100:00080000:2.0:1787830446.485155:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830446.485159:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787830446.485165:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000008:00100000:2.0:1787830446.485165:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830446.485173:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518265216:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830446.485180:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518265600/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830446.485190:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830446.485193:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830446.485196:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830446.485200:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518265344:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787830446.485203:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3452 00000100:00080000:2.0:1787830446.485209:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830446.485214:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830446.485219:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830446.485229:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518265728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830446.485242:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787830446.485255:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830446.485259:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830446.485261:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830446.485265:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3452 00000100:00100000:0.0:1787830446.485269:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830446/real 1787830446] req@ffff8b2e8801c780 x1874672518265344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830446.485269:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830446.485276:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830446.485284:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830446.485297:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518265856/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830446.485300:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518265344/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830446.485312:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830446.485318:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830446.485321:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830446.485328:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3452 00000100:00080000:0.0:1787830446.485329:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830446.485332:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830446.485336:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:2.0:1787830446.485338:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830446.485344:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518265344:192.168.203.101@tcp:38:kworker.0 00000008:00100000:2.0:1787830446.485345:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830446.485356:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518265472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830446.485359:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518265984/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830446.485384:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830446.485398:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830446/real 1787830446] req@ffff8b2e8801f0c0 x1874672518265472/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830446.485410:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518265472/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830446.485418:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830446.485421:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830446.485424:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518265472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830446.485430:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518265600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830446.485448:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830446.485459:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830446/real 1787830446] req@ffff8b2e8801d680 x1874672518265600/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830446.485470:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518265600/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830446.485478:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830446.485481:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830446.485485:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518265600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830446.485500:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518265728:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830446.485519:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830446.485531:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830446/real 1787830446] req@ffff8b2e8801c3c0 x1874672518265728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830446.485542:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518265728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830446.485550:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830446.485553:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830446.485556:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518265728:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830446.485562:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518265856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830446.485580:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830446.485590:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830446/real 1787830446] req@ffff8b2e8801cb40 x1874672518265856/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830446.485613:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518265856/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830446.485622:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830446.485625:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830446.485628:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518265856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830446.485633:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518265984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830446.485693:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830446.485704:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830446/real 1787830446] req@ffff8b2e8801de00 x1874672518265984/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830446.485715:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518265984/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830471 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830446.485723:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830446.485726:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830446.485729:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518265984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830446.804775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830447.316945:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830447.317448:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830447.508927:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830447.510627:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830447.829126:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830448.340776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830448.340873:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830448.532792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830448.533263:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830448.852870:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830449.364787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830449.364877:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830449.556815:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830449.556925:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830449.881126:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830450.388769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830450.388876:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830450.580791:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830450.581016:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830450.900794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830451.412884:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830451.412886:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830451.604792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787830451.605010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830451.605015:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830451.605017:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830451.605025:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3458 00000100:00080000:2.0:1787830451.605034:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830451.605039:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830451.605076:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518266112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830451.605108:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830451.605112:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830451.605114:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830451.605118:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3458 00000100:00080000:2.0:1787830451.605122:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830451.605126:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830451.605135:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830451.605146:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518266240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830451.605153:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830451.605155:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830451.605156:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830451.605158:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3458 00000100:00080000:2.0:1787830451.605161:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830451.605162:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830451.605165:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830451.605171:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518266368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830451.605176:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830451.605178:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830451.605179:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830451.605180:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3458 00000100:00080000:2.0:1787830451.605182:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830451.605183:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830451.605186:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830451.605191:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518266496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830451.605196:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830451.605202:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830451.605203:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830451.605205:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3458 00000100:00080000:2.0:1787830451.605207:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830451.605209:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830451.605211:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830451.605216:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518266624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830451.605225:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830451.605228:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830451.605230:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830451.605241:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3458 00000100:00080000:2.0:1787830451.605245:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830451.605249:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830451.605254:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830451.605257:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518266112:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830451.605259:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830451.605267:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518266752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830451.605279:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830451.605282:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830451.605284:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830451.605289:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3458 00000100:00080000:2.0:1787830451.605292:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830451.605297:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830451.605301:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830451.605311:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518266880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830451.605326:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830451.605377:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830451/real 1787830451] req@ffff8b2e8801de00 x1874672518266112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830451.605394:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518266112/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830451.605403:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830451.605407:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830451.605411:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518266112:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830451.605418:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518266240:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830451.605440:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830451.605452:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830451/real 1787830451] req@ffff8b2e8801cb40 x1874672518266240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830451.605463:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518266240/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830451.605472:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830451.605475:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830451.605478:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518266240:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830451.605484:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518266368:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830451.605503:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830451.605520:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830451/real 1787830451] req@ffff8b2e8801c3c0 x1874672518266368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830451.605531:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518266368/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830451.605546:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830451.605550:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830451.605553:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518266368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830451.605559:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518266496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830451.605577:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830451.605588:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830451/real 1787830451] req@ffff8b2e8801d680 x1874672518266496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830451.605605:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518266496/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830451.605613:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830451.605616:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830451.605619:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518266496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830451.605624:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518266624:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830451.605642:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830451.605681:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830451/real 1787830451] req@ffff8b2e8801f0c0 x1874672518266624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830451.605696:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518266624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830451.605704:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830451.605707:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830451.605710:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518266624:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830451.605735:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518266752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830451.605750:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830451.605757:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830451/real 1787830451] req@ffff8b2e8801c780 x1874672518266752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830451.605775:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518266752/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830451.605782:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830451.605785:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830451.605788:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518266752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830451.605793:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518266880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830451.605811:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830451.605821:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830451/real 1787830451] req@ffff8b2e8801ed00 x1874672518266880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830451.605828:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518266880/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830476 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830451.605833:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830451.605835:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830451.605837:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518266880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830451.924797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830452.436770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830452.437250:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830452.628781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830452.628781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830452.949979:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830453.460771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830453.460865:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830453.652824:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830453.652834:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830453.972784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830454.484931:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830454.485258:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830454.676801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830454.677097:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830454.996781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830455.508823:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830455.508848:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830455.700851:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830455.700851:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830456.020799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830456.532767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830456.532870:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830456.724752:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830456.724752:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787830456.725002:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830456.725012:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830456.725016:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830456.725025:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3463 00000100:00080000:2.0:1787830456.725036:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830456.725043:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830456.725080:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518267008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830456.725110:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830456.725113:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830456.725115:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830456.725119:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3463 00000100:00080000:2.0:1787830456.725123:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830456.725126:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830456.725136:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830456.725148:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518267136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830456.725158:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830456.725161:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830456.725163:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830456.725167:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3463 00000100:00080000:2.0:1787830456.725170:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830456.725175:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830456.725178:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518267008:192.168.203.101@tcp:250:kworker.0 00000008:00100000:2.0:1787830456.725181:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830456.725193:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518267264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830456.725202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830456.725205:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830456.725207:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830456.725210:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3463 00000100:00080000:2.0:1787830456.725213:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830456.725217:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830456.725221:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830456.725232:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518267392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830456.725241:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830456.725243:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830456.725245:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830456.725249:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3463 00000100:00080000:2.0:1787830456.725253:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830456.725256:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830456.725260:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830456.725269:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518267520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830456.725278:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830456.725281:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00100000:0.0:1787830456.725284:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518267136:192.168.203.101@tcp:38:kworker.0 00000100:00080000:2.0:1787830456.725285:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830456.725292:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3463 00000100:00080000:2.0:1787830456.725297:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830456.725300:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830456.725305:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830456.725315:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518267648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830456.725324:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830456.725326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518267264:192.168.203.101@tcp:8:kworker.0 00000100:00080000:2.0:1787830456.725328:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830456.725332:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830456.725337:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3463 00000100:00080000:2.0:1787830456.725341:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830456.725344:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830456.725349:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830456.725355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518267392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830456.725363:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518267776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830456.725380:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518267520:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830456.725406:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518267648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830456.725424:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518267776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830457.044795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830457.556808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830457.556920:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830457.748788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830457.748868:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830458.068724:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830458.580775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830458.580862:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830458.772773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830458.772836:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830459.092885:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830459.604811:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830459.604875:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830459.796896:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830459.796929:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:3.0:1787830459.797169:0:2336:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:3.0:1787830459.797213:0:2336:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787830459.797227:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830459.797265:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830459.797275:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830459.797283:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830459.797290:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830459.797296:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830459.797303:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787830459.797734:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830456/real 1787830459] req@ffff8b2e8801e580 x1874672518267648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830459.797758:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518267648/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830459.797770:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830459.797775:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830459.797779:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518267648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830459.797786:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830456/real 1787830459] req@ffff8b2e8801e940 x1874672518267776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830459.797797:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518267776/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830459.797803:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830459.797807:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830459.797810:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518267776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830459.797815:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830456/real 1787830459] req@ffff8b2e8801f480 x1874672518267264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830459.797826:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518267264/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830459.797833:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830459.797836:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830459.797838:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518267264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830459.797844:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830456/real 1787830459] req@ffff8b2e8801cf00 x1874672518267392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830459.797853:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518267392/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830459.797860:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830459.797863:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830459.797866:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518267392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830459.797871:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830456/real 1787830459] req@ffff8b2e8801c000 x1874672518267520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830459.797883:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518267520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830459.797890:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830459.797893:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830459.797897:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518267520:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830459.797902:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830456/real 1787830459] req@ffff8b2e8801da40 x1874672518267008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830459.798362:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518267008/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830459.798372:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830459.798375:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830459.798379:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518267008:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830459.798385:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830456/real 1787830459] req@ffff8b2e8801fc00 x1874672518267136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830459.798397:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518267136/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830481 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830459.798405:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830459.798409:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830459.798411:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518267136:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830460.116789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830460.628771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830460.628812:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830460.820762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830460.820834:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830461.140802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830461.652782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830461.653932:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830461.844766:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830461.844776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830461.844778:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830461.844785:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830461.844795:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3468 00000100:00080000:3.0:1787830461.844805:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830461.844811:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830461.844832:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830461.844854:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518267904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830461.844882:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830461.844885:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830461.844887:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830461.844911:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3468 00000100:00080000:3.0:1787830461.844915:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830461.844920:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830461.844929:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830461.844937:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518267904:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830461.844945:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518268032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830461.844958:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830461.844962:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830461.844964:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830461.844968:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3468 00000100:00080000:3.0:1787830461.844972:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830461.844976:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830461.844981:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830461.844993:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518268160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830461.844997:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830461.845008:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830461.845011:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830461.845013:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830461.845018:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3468 00000100:00080000:3.0:1787830461.845022:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830461.845026:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830461.845031:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830461.845038:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830461/real 1787830461] req@ffff8b2e88039a40 x1874672518267904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830461.845044:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518268288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830461.845061:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830461.845064:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518267904/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830461.845066:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830461.845069:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830461.845078:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3468 00000100:00080000:0.0:1787830461.845082:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830461.845083:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830461.845089:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787830461.845092:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830461.845097:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518267904:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787830461.845100:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830461.845114:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518268416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830461.845122:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518268032:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830461.845130:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830461.845135:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830461.845143:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830461.845148:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3468 00000100:00080000:3.0:1787830461.845152:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787830461.845153:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830461.845160:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830461.845166:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830461.845171:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830461/real 1787830461] req@ffff8b2e8803a580 x1874672518268032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830461.845180:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518268544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830461.845188:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518268032/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830461.845196:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830461.845203:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830461.845203:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830461.845208:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830461.845211:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787830461.845216:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3468 00000100:00100000:0.0:1787830461.845219:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518268032:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830461.845222:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830461.845229:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518268160:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830461.845230:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830461.845239:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830461.845258:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830461.845261:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518268672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830461.845272:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830461/real 1787830461] req@ffff8b2e8803b480 x1874672518268160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830461.845284:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518268160/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830461.845291:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830461.845294:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830461.845296:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518268160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830461.845302:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518268288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830461.845318:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830461.845328:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830461/real 1787830461] req@ffff8b2e8803a1c0 x1874672518268288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830461.845344:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518268288/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830461.845351:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830461.845354:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830461.845357:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518268288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830461.845379:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518268416:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830461.845404:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830461.845415:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830461/real 1787830461] req@ffff8b2e88039e00 x1874672518268416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830461.845426:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518268416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830461.845434:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830461.845437:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830461.845439:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518268416:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830461.845444:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518268544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830461.845462:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830461.845472:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830461/real 1787830461] req@ffff8b2e8803b840 x1874672518268544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830461.845483:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518268544/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830461.845500:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830461.845504:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830461.845506:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518268544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830461.847193:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518268672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830461.847253:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830461.847278:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830461/real 1787830461] req@ffff8b2e88039680 x1874672518268672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830461.847293:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518268672/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830486 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830461.847302:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830461.847306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830461.847309:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518268672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830462.164877:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830462.676781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830462.676820:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830462.868762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830462.868877:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830463.188788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830463.700944:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830463.701139:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830463.892800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830463.892921:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830464.212794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830464.724775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830464.725758:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830464.916787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830464.920012:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830465.236827:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830465.748753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830465.748958:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830465.940814:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830465.940930:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830466.260790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830466.772837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830466.773040:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830466.964919:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787830466.966975:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830466.966982:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830466.966985:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830466.966993:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3473 00000100:00100000:0.0:1787830466.966994:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787830466.967005:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830466.967013:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830466.967053:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518268800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830466.967086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830466.967089:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830466.967092:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830466.967096:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3473 00000100:00080000:2.0:1787830466.967100:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830466.967103:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830466.967112:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830466.967125:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518268928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830466.967135:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830466.967138:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830466.967139:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830466.967142:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3473 00000100:00080000:2.0:1787830466.967146:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830466.967149:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830466.967153:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830466.967162:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518269056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830466.967169:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830466.967172:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830466.967173:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830466.967176:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3473 00000100:00080000:2.0:1787830466.967179:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830466.967181:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830466.967184:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830466.967191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518269184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830466.967199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830466.967202:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830466.967203:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830466.967206:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3473 00000100:00080000:2.0:1787830466.967208:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830466.967211:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830466.967214:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830466.967220:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518269312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830466.967227:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830466.967229:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830466.967230:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830466.967233:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3473 00000100:00080000:2.0:1787830466.967236:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830466.967239:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830466.967242:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830466.967251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518269440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830466.967258:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830466.967261:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830466.967263:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830466.967266:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3473 00000100:00080000:2.0:1787830466.967269:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830466.967272:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830466.967276:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830466.967283:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518269568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830466.967354:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518268800:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830466.967419:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830466.967458:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830466/real 1787830466] req@ffff8b2e8801fc00 x1874672518268800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830466.967473:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518268800/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830466.967488:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830466.967492:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830466.967495:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518268800:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830466.967503:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518268928:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830466.967531:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830466.967546:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830466/real 1787830466] req@ffff8b2e8801da40 x1874672518268928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830466.967558:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518268928/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830466.967566:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830466.967569:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830466.967572:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518268928:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830466.967577:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518269056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830466.967595:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830466.967606:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830466/real 1787830466] req@ffff8b2e8801c000 x1874672518269056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830466.967616:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518269056/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830466.967624:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830466.967626:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830466.967629:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518269056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830466.967635:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518269184:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830466.967672:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830466.967683:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830466/real 1787830466] req@ffff8b2e8801cf00 x1874672518269184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830466.967694:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518269184/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830466.967701:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830466.967703:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830466.967707:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518269184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830466.967712:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518269312:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830466.967730:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830466.967741:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830466/real 1787830466] req@ffff8b2e8801f480 x1874672518269312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830466.967751:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518269312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830466.967758:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830466.967761:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830466.967764:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518269312:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830466.967768:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518269440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830466.967792:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830466.967802:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830466/real 1787830466] req@ffff8b2e8801e940 x1874672518269440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830466.967814:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518269440/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830466.967822:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830466.967825:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830466.967828:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518269440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830466.967833:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518269568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830466.967850:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830466.967873:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830466/real 1787830466] req@ffff8b2e8801e580 x1874672518269568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830466.967884:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518269568/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830491 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830466.967891:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830466.967894:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830466.967897:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518269568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830467.284761:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830467.796763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830467.798459:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830467.989206:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830467.989328:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830468.309069:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830468.820770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830468.821214:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830469.012821:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830469.012913:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830469.332918:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830469.844771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830469.844807:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830470.036779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830470.036779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830470.356790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830470.868756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830470.868860:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830471.060774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830471.060774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830471.380787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830471.892772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830471.892892:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830472.084800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830472.084800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830472.084875:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830472.084881:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830472.084885:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830472.084894:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3478 00000100:00080000:3.0:1787830472.084905:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830472.084912:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830472.084949:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518269696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830472.084970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830472.084973:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830472.084974:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830472.084977:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3478 00000100:00080000:3.0:1787830472.084979:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830472.084982:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830472.084990:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830472.084999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518269824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830472.085005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830472.085007:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830472.085009:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830472.085011:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3478 00000100:00080000:3.0:1787830472.085013:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830472.085015:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830472.085017:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830472.085022:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518269952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830472.085026:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830472.085028:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830472.085029:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830472.085031:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3478 00000100:00080000:3.0:1787830472.085032:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830472.085034:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830472.085036:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830472.085041:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518270080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830472.085046:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830472.085047:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830472.085048:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830472.085051:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3478 00000100:00080000:3.0:1787830472.085053:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830472.085055:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830472.085057:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830472.085063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518270208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830472.085069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830472.085071:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518269696:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830472.085072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830472.085073:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830472.085080:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3478 00000100:00080000:3.0:1787830472.085082:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830472.085084:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830472.085086:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830472.085092:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518270336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830472.085096:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830472.085098:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830472.085099:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830472.085101:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3478 00000100:00080000:3.0:1787830472.085103:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830472.085105:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830472.085108:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830472.085113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518270464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830472.085134:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830472.085178:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830472/real 1787830472] req@ffff8b2e880392c0 x1874672518269696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830472.085196:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518269696/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830472.085213:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830472.085217:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830472.085221:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518269696:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830472.085228:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518269824:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830472.085251:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830472.085264:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830472/real 1787830472] req@ffff8b2e88039680 x1874672518269824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830472.085277:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518269824/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830472.085286:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830472.085289:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830472.085291:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518269824:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830472.085297:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518269952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830472.085318:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830472.085330:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830472/real 1787830472] req@ffff8b2e8803b840 x1874672518269952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830472.085352:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518269952/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830472.085360:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830472.085363:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830472.085366:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518269952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830472.085372:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518270080:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830472.085392:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830472.085403:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830472/real 1787830472] req@ffff8b2e88039e00 x1874672518270080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830472.085415:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518270080/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830472.085423:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830472.085426:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830472.085429:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518270080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830472.085454:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518270208:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830472.085475:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830472.085487:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830472/real 1787830472] req@ffff8b2e8803a1c0 x1874672518270208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830472.085499:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518270208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830472.085508:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830472.085511:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830472.085514:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518270208:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830472.085520:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518270336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830472.085540:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830472.085550:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830472/real 1787830472] req@ffff8b2e8803b480 x1874672518270336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830472.085560:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518270336/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830472.085567:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830472.085570:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830472.085573:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518270336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830472.085578:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518270464:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830472.085598:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830472.085607:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830472/real 1787830472] req@ffff8b2e8803a580 x1874672518270464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830472.085617:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518270464/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830497 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830472.085624:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830472.085627:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830472.085630:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518270464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830472.404796:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830472.916958:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830472.917029:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830473.108786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830473.108786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830473.428768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830473.940794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830473.941351:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830474.132802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830474.132876:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830474.453032:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830474.964764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830474.964830:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830475.156753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830475.156867:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830475.476822:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830475.988845:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830475.989175:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830476.180782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830476.180785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830476.500930:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830477.012831:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830477.014318:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830477.204735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830477.204775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787830477.204869:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830477.204874:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830477.204877:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830477.204885:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3483 00000100:00080000:2.0:1787830477.204900:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830477.204907:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830477.204942:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518270592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830477.204970:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830477.204974:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830477.204976:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830477.204980:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3483 00000100:00080000:2.0:1787830477.204983:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830477.204987:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830477.204996:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830477.205007:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518270720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830477.205016:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830477.205019:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830477.205020:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830477.205024:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3483 00000100:00080000:2.0:1787830477.205027:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830477.205030:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830477.205034:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830477.205042:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518270848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830477.205050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830477.205053:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830477.205055:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830477.205058:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3483 00000100:00080000:2.0:1787830477.205061:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830477.205064:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830477.205067:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830477.205076:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518270976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830477.205084:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830477.205087:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830477.205088:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830477.205092:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3483 00000100:00080000:2.0:1787830477.205095:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830477.205098:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830477.205102:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830477.205111:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518271104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830477.205118:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830477.205121:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830477.205123:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830477.205126:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3483 00000100:00080000:2.0:1787830477.205129:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830477.205132:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830477.205136:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830477.205144:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518271232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830477.205151:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830477.205154:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830477.205156:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830477.205159:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3483 00000100:00080000:2.0:1787830477.205161:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830477.205164:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830477.205168:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830477.205176:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518271360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830477.205364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518270592:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830477.205430:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830477.205470:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830477/real 1787830477] req@ffff8b2e8801ed00 x1874672518270592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830477.205486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518270592/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830477.205500:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830477.205504:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830477.205508:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518270592:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830477.205514:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518270720:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830477.205534:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830477.205546:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830477/real 1787830477] req@ffff8b2e8801c780 x1874672518270720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830477.205558:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518270720/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830477.205578:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830477.205581:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830477.205584:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518270720:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830477.205590:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518270848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830477.205608:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830477.205618:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830477/real 1787830477] req@ffff8b2e8801f0c0 x1874672518270848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830477.205629:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518270848/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830477.205637:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830477.205640:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830477.205643:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518270848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830477.205687:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518270976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830477.205706:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830477.205717:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830477/real 1787830477] req@ffff8b2e8801d680 x1874672518270976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830477.205728:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518270976/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830477.205736:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830477.205739:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830477.205741:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518270976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830477.205746:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518271104:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830477.205763:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830477.205772:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830477/real 1787830477] req@ffff8b2e8801c3c0 x1874672518271104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830477.205783:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518271104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830477.205803:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830477.205806:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830477.205809:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518271104:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830477.205814:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518271232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830477.205834:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830477.205845:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830477/real 1787830477] req@ffff8b2e8801cb40 x1874672518271232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830477.205855:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518271232/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830477.205863:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830477.205865:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830477.205868:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518271232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830477.205873:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518271360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830477.205896:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830477.205905:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830477/real 1787830477] req@ffff8b2e8801de00 x1874672518271360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830477.205916:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518271360/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830502 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830477.205923:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830477.205926:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830477.205929:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518271360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830477.524831:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830478.036991:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830478.037587:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830478.228770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830478.228884:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830478.548995:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830479.060854:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830479.060861:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830479.252803:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830479.252877:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830479.572902:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830480.084765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830480.084843:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830480.277132:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830480.277436:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830480.596820:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830481.108801:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830481.108840:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830481.300786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830481.300786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830481.620797:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830482.132795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830482.132831:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830482.324800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830482.324838:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830482.324894:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830482.324899:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830482.324903:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830482.324923:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3488 00000100:00080000:0.0:1787830482.324934:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830482.324941:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830482.324988:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518271488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830482.325027:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830482.325031:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830482.325033:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830482.325037:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3488 00000100:00080000:0.0:1787830482.325041:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830482.325045:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830482.325054:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830482.325066:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518271616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830482.325075:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830482.325077:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830482.325079:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830482.325082:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3488 00000100:00080000:0.0:1787830482.325085:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830482.325088:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830482.325093:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830482.325102:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518271744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830482.325110:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830482.325113:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830482.325115:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830482.325118:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3488 00000100:00080000:0.0:1787830482.325122:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830482.325125:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830482.325130:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830482.325140:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518271872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830482.325167:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518271488:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830482.325244:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830482.325459:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830482.325461:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830482.325462:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830482.325466:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3488 00000100:00080000:0.0:1787830482.325470:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830482.325473:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830482.325478:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830482.325487:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830482/real 1787830482] req@ffff8b2e880421c0 x1874672518271488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830482.325502:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518272000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830482.325509:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518271488/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830482.325518:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830482.325525:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830482.325526:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830482.325527:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830482.325535:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3488 00000100:00080000:0.0:1787830482.325538:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830482.325539:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:0.0:1787830482.325544:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830482.325548:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518271488:192.168.203.101@tcp:250:kworker.0 00000008:00100000:0.0:1787830482.325550:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830482.325560:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518271616:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830482.325563:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518272128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830482.325577:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830482.325579:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830482.325581:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830482.325584:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3488 00000100:00080000:0.0:1787830482.325588:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830482.325590:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787830482.325595:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:0.0:1787830482.325597:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830482.325607:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518272256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830482.325611:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830482/real 1787830482] req@ffff8b2e88041a40 x1874672518271616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830482.325626:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518271616/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830482.325633:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830482.325637:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830482.325640:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518271616:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830482.325672:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518271744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830482.325696:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830482.325707:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830482/real 1787830482] req@ffff8b2e88040f00 x1874672518271744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830482.325723:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518271744/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830482.325730:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830482.325733:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830482.325736:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518271744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830482.325783:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518271872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830482.325804:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830482.325815:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830482/real 1787830482] req@ffff8b2e88042940 x1874672518271872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830482.325825:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518271872/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830482.325833:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830482.325837:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830482.325840:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518271872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830482.325845:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518272000:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830482.325863:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830482.325872:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830482/real 1787830482] req@ffff8b2e88043480 x1874672518272000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830482.325884:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518272000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830482.325898:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830482.325901:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830482.325904:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518272000:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830482.325909:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518272128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830482.325924:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830482.325933:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830482/real 1787830482] req@ffff8b2e880430c0 x1874672518272128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830482.325943:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518272128/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830482.325951:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830482.325953:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830482.325956:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518272128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830482.325962:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518272256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830482.325980:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830482.325990:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830482/real 1787830482] req@ffff8b2e88040000 x1874672518272256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830482.325999:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518272256/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830507 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830482.326006:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830482.326008:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830482.326011:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518272256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830482.644808:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830483.156922:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830483.158077:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830483.348807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830483.348807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830483.668946:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830484.180750:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830484.180988:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830484.372801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830484.372837:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830484.692813:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830485.204787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830485.205900:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830485.396807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830485.396808:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830485.717604:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830486.228795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830486.228853:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830486.420801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830486.421361:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830486.740900:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830487.252807:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830487.252891:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830487.444755:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830487.444796:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830487.444810:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830487.444814:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830487.444822:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3493 00000100:00080000:3.0:1787830487.444831:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830487.444837:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830487.444891:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518272384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830487.444978:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830487.444983:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518272384:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830487.445430:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830487.445434:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830487.445436:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830487.445441:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3493 00000800:00000100:0.0:1787830487.445443:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830487.445445:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830487.445448:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830487.445458:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830487.445471:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518272512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830487.445480:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830487.445483:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830487.445485:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830487.445488:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3493 00000100:00100000:0.0:1787830487.445490:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830487/real 1787830487] req@ffff8b2e88039a40 x1874672518272384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830487.445492:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830487.445495:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830487.445500:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830487.445509:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518272640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830487.445517:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830487.445520:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830487.445521:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830487.445525:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3493 00000100:00100000:0.0:1787830487.445526:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518272384/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830487.445528:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830487.445531:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787830487.445535:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787830487.445535:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787830487.445539:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830487.445544:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518272768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830487.445552:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830487.445553:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518272384:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830487.445556:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830487.445557:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830487.445560:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3493 00000100:00080000:3.0:1787830487.445563:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830487.445566:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830487.445570:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830487.445573:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518272512:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830487.445581:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518272896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830487.445595:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830487.445598:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830487.445600:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787830487.445603:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830487.445603:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3493 00000100:00080000:3.0:1787830487.445606:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830487.445609:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830487.445613:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830487.445622:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518273024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830487.445625:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830487/real 1787830487] req@ffff8b2e8803b0c0 x1874672518272512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830487.445631:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830487.445634:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830487.445636:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830487.445639:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518272512/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830487.445639:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3493 00000100:00080000:3.0:1787830487.445642:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830487.445668:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830487.445674:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787830487.445676:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830487.445679:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830487.445683:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518272512:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830487.445683:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518273152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830487.445700:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518272640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830487.445743:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830487.445753:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830487/real 1787830487] req@ffff8b2e88038f00 x1874672518272640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830487.445761:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518272640/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830487.445782:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830487.445785:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830487.445787:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518272640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830487.445791:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518272768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830487.445815:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830487.445822:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830487/real 1787830487] req@ffff8b2e880383c0 x1874672518272768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830487.445828:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518272768/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830487.445846:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830487.445848:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830487.445850:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518272768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830487.445861:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518272896:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830487.445884:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830487.445891:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830487/real 1787830487] req@ffff8b2e88039a40 x1874672518272896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830487.445905:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518272896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830487.445910:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830487.445912:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830487.445914:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518272896:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830487.445926:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518273024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830487.445937:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830487.445943:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830487/real 1787830487] req@ffff8b2e8803a580 x1874672518273024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830487.445950:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518273024/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830487.445974:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830487.445975:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830487.445978:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518273024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830487.445982:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518273152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830487.446776:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830487.446795:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830487/real 1787830487] req@ffff8b2e8803b480 x1874672518273152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830487.446809:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518273152/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830512 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830487.446817:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830487.446820:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830487.446824:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518273152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830487.764829:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830488.276912:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830488.277247:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830488.468815:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830488.468815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830488.789289:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830489.300798:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830489.300878:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830489.492851:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830489.492922:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830489.812895:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830490.324782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830490.324846:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830490.517391:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830490.517391:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830490.836764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830491.348873:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830491.348942:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830491.540749:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830491.541005:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830491.861119:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830492.372788:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830492.374337:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830492.564777:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830492.564780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830492.564780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830492.564789:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830492.564795:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830492.564805:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3498 00000100:00080000:3.0:1787830492.564815:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830492.564822:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830492.564858:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518273280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830492.564882:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830492.564885:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830492.564887:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830492.564891:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3498 00000100:00080000:3.0:1787830492.564894:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830492.564898:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830492.564908:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830492.564918:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518273280:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830492.564921:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518273408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830492.564932:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830492.564936:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830492.564938:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830492.564941:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3498 00000100:00080000:3.0:1787830492.564944:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830492.564947:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830492.564952:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830492.564964:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518273536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830492.564978:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787830492.564983:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830492.564984:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830492.564988:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830492.564992:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3498 00000100:00080000:3.0:1787830492.564995:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830492.564999:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830492.565003:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830492.565014:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518273664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830492.565021:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830492.565024:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830492.565026:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830492.565032:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3498 00000100:00100000:0.0:1787830492.565033:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830492/real 1787830492] req@ffff8b2e8803a1c0 x1874672518273280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830492.565035:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830492.565042:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830492.565050:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830492.565060:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518273280/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830492.565064:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518273792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830492.565079:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830492.565084:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830492.565086:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830492.565089:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3498 00000100:00080000:3.0:1787830492.565092:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830492.565096:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787830492.565097:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787830492.565101:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787830492.565105:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830492.565111:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518273280:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830492.565114:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518273920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830492.565129:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830492.565132:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830492.565133:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830492.565136:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3498 00000100:00080000:3.0:1787830492.565139:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830492.565143:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518273408:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830492.565144:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830492.565152:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830492.565164:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518274048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830492.565185:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830492.565199:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830492/real 1787830492] req@ffff8b2e88039e00 x1874672518273408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830492.565212:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518273408/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830492.565220:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830492.565223:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830492.565226:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518273408:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830492.565232:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518273536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830492.565252:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830492.565263:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830492/real 1787830492] req@ffff8b2e8803b840 x1874672518273536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830492.565284:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518273536/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830492.565292:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830492.565295:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830492.565299:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518273536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830492.565304:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518273664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830492.565323:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830492.565333:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830492/real 1787830492] req@ffff8b2e88039680 x1874672518273664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830492.565344:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518273664/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830492.565352:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830492.565355:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830492.565359:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518273664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830492.565364:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518273792:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830492.565382:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830492.565393:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830492/real 1787830492] req@ffff8b2e880392c0 x1874672518273792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830492.565404:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518273792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830492.565411:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830492.565414:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830492.565417:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518273792:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830492.565423:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518273920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830492.565454:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830492.565465:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830492/real 1787830492] req@ffff8b2e8803b480 x1874672518273920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830492.565476:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518273920/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830492.565488:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830492.565491:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830492.565495:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518273920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830492.565511:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518274048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830492.565530:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830492.565541:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830492/real 1787830492] req@ffff8b2e8803a580 x1874672518274048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830492.565553:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518274048/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830517 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830492.565568:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830492.565571:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830492.565574:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518274048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830492.885177:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830493.396897:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830493.397443:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830493.588782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830493.589889:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830493.909206:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830494.420779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830494.421101:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830494.612934:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830494.614051:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830494.932821:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830495.444805:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830495.444827:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830495.636802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830495.637007:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830495.956780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830496.468933:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830496.469142:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830496.660825:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830496.660832:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830496.980854:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830497.492776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830497.492837:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830497.684802:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830497.685415:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830497.685421:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830497.685425:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830497.685433:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3504 00000100:00080000:3.0:1787830497.685443:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830497.685450:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830497.685486:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518274176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830497.685518:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830497.685523:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830497.685524:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830497.685529:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3504 00000100:00080000:3.0:1787830497.685533:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830497.685536:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830497.685545:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830497.685557:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518274304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830497.685568:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830497.685571:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830497.685573:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830497.685576:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3504 00000100:00080000:3.0:1787830497.685580:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830497.685585:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518274176:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830497.685586:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830497.685594:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830497.685608:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518274432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830497.685619:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830497.685622:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830497.685624:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830497.685628:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3504 00000100:00080000:3.0:1787830497.685631:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830497.685635:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830497.685640:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830497.685672:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830497.685694:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518274560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830497.685704:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830497.685707:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830497.685709:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830497.685714:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3504 00000100:00080000:3.0:1787830497.685718:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830497.685725:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830497/real 1787830497] req@ffff8b2e88039a40 x1874672518274176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830497.685736:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830497.685743:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830497.685756:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518274688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830497.685765:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830497.685768:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830497.685770:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830497.685773:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3504 00000100:00080000:3.0:1787830497.685776:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830497.685779:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830497.685783:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830497.685792:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518274816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830497.685801:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518274176/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830497.685803:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830497.685811:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830497.685815:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830497.685823:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3504 00000100:00080000:3.0:1787830497.685830:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830497.685833:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830497.685836:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830497.685843:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830497.685854:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518274944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830497.685908:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830497.685915:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518274176:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830497.685923:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518274304:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830497.685948:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830497.685961:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830497/real 1787830497] req@ffff8b2e880383c0 x1874672518274304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830497.685973:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518274304/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830497.685982:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830497.685985:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830497.685988:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518274304:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830497.686096:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518274432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830497.686120:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830497.686131:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830497/real 1787830497] req@ffff8b2e88038f00 x1874672518274432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830497.686143:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518274432/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830497.686152:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830497.686155:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830497.686158:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518274432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830497.686165:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518274560:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830497.686184:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830497.686195:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830497/real 1787830497] req@ffff8b2e8803b0c0 x1874672518274560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830497.686208:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518274560/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830497.686216:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830497.686219:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830497.686223:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518274560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830497.686228:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518274688:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830497.686246:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830497.686257:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830497/real 1787830497] req@ffff8b2e8803a580 x1874672518274688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830497.686269:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518274688/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830497.686277:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830497.686280:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830497.686283:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518274688:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830497.686289:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518274816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830497.686307:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830497.686318:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830497/real 1787830497] req@ffff8b2e8803b480 x1874672518274816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830497.686329:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518274816/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830497.686337:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830497.686341:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830497.686344:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518274816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830497.686349:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518274944:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830497.686367:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830497.686377:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830497/real 1787830497] req@ffff8b2e880392c0 x1874672518274944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830497.686388:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518274944/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830522 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830497.686396:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830497.686399:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830497.686402:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518274944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830497.690030:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830498.004813:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830498.516764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830498.516889:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830498.708784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830498.708973:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830499.030464:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830499.540808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830499.540967:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830499.732779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830499.732779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830500.052787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830500.565465:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830500.565715:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830500.756874:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830500.758100:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830501.076794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830501.589110:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830501.589295:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830501.780773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830501.780773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830502.100875:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830502.613129:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830502.613529:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830502.804810:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830502.804892:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830502.804930:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830502.804936:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830502.804940:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830502.804949:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3509 00000100:00080000:3.0:1787830502.804958:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830502.804964:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830502.805003:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518275072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830502.805030:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830502.805033:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830502.805035:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830502.805039:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3509 00000100:00080000:3.0:1787830502.805041:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830502.805044:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830502.805053:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830502.805063:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518275200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830502.805072:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830502.805075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830502.805077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830502.805080:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3509 00000100:00080000:3.0:1787830502.805083:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830502.805086:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830502.805090:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830502.805102:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518275328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830502.805111:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830502.805114:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830502.805120:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830502.805124:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3509 00000100:00080000:3.0:1787830502.805127:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830502.805130:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830502.805135:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830502.805145:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518275456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830502.805154:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830502.805157:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830502.805159:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830502.805163:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3509 00000100:00080000:3.0:1787830502.805166:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830502.805173:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830502.805180:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830502.805191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518275584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830502.805199:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830502.805202:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830502.805203:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830502.805208:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3509 00000100:00080000:3.0:1787830502.805211:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830502.805214:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830502.805217:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830502.805224:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518275712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830502.805231:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830502.805234:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830502.805235:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830502.805238:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3509 00000100:00080000:3.0:1787830502.805240:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830502.805242:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830502.805245:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830502.805251:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518275840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830502.805312:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518275072:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830502.805367:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830502.805400:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830502/real 1787830502] req@ffff8b2e88039680 x1874672518275072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830502.805414:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518275072/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830502.805422:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830502.805426:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830502.805429:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518275072:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830502.805435:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518275200:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830502.805451:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830502.805460:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830502/real 1787830502] req@ffff8b2e8803b840 x1874672518275200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830502.805469:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518275200/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830502.805476:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830502.805479:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830502.805482:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518275200:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830502.805488:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518275328:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830502.805502:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830502.805508:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830502/real 1787830502] req@ffff8b2e88039e00 x1874672518275328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830502.805515:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518275328/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830502.805524:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830502.805526:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830502.805527:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518275328:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830502.805530:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518275456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830502.805538:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830502.805543:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830502/real 1787830502] req@ffff8b2e8803a1c0 x1874672518275456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830502.805549:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518275456/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830502.805554:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830502.805555:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830502.805557:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518275456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830502.805560:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518275584:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830502.805568:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830502.805573:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830502/real 1787830502] req@ffff8b2e880392c0 x1874672518275584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830502.805579:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518275584/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830502.805583:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830502.805584:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830502.805587:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518275584:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830502.805590:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518275712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830502.805598:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830502.805602:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830502/real 1787830502] req@ffff8b2e8803b480 x1874672518275712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830502.805608:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518275712/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830502.805625:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830502.805628:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830502.805630:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518275712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830502.805633:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518275840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830502.805670:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830502.805681:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830502/real 1787830502] req@ffff8b2e8803a580 x1874672518275840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830502.805692:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518275840/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830527 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830502.805700:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830502.805703:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830502.805706:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518275840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830503.125114:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830503.637621:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830503.638923:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830503.828785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830503.828828:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830504.149211:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830504.660778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830504.661046:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830504.852770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830504.853506:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830505.173123:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830505.684782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830505.684928:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830505.876808:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830505.876935:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830506.196790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830506.708715:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830506.709928:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830506.900786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830506.900898:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830507.220840:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830507.732775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830507.732958:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830507.924761:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830507.924768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830507.924773:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830507.924777:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830507.924787:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3514 00000100:00100000:2.0:1787830507.924791:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830507.924797:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830507.924805:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830507.924840:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518275968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830507.924865:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830507.924868:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830507.924870:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830507.924873:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3514 00000100:00080000:3.0:1787830507.924877:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830507.924880:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830507.924890:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830507.924901:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518276096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830507.924910:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830507.924913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830507.924915:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830507.924918:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3514 00000100:00080000:3.0:1787830507.924920:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830507.924924:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830507.924928:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830507.924935:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518276224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830507.924944:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830507.924947:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830507.924948:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830507.924951:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3514 00000100:00080000:3.0:1787830507.924954:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830507.924956:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830507.924959:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830507.924967:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518276352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830507.924974:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830507.924977:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830507.924978:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830507.924981:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3514 00000100:00080000:3.0:1787830507.924983:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830507.924985:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830507.924989:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830507.924997:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518276480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830507.925005:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830507.925008:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830507.925009:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830507.925012:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3514 00000100:00080000:3.0:1787830507.925015:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830507.925017:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830507.925021:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830507.925027:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518276608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830507.925034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830507.925037:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830507.925038:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830507.925041:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3514 00000100:00080000:3.0:1787830507.925043:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830507.925046:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830507.925049:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830507.925056:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518276736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830507.925104:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518275968:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830507.925159:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830507.925199:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830507/real 1787830507] req@ffff8b2e8803b0c0 x1874672518275968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830507.925226:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518275968/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830507.925236:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830507.925240:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830507.925244:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518275968:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830507.925251:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518276096:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830507.926012:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830507.926028:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830507/real 1787830507] req@ffff8b2e88038f00 x1874672518276096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830507.926042:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518276096/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830507.926050:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830507.926054:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830507.926057:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518276096:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830507.926074:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518276224:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830507.926096:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830507.926108:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830507/real 1787830507] req@ffff8b2e880383c0 x1874672518276224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830507.926119:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518276224/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830507.926126:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830507.926128:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830507.926131:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518276224:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830507.926137:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518276352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830507.926153:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830507.926163:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830507/real 1787830507] req@ffff8b2e88039a40 x1874672518276352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830507.926173:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518276352/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830507.926181:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830507.926184:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830507.926187:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518276352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830507.926192:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518276480:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830507.926220:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830507.926230:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830507/real 1787830507] req@ffff8b2e8803a580 x1874672518276480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830507.926240:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518276480/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830507.926249:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830507.926252:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830507.926255:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518276480:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830507.926260:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518276608:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830507.926287:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830507.926297:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830507/real 1787830507] req@ffff8b2e8803b480 x1874672518276608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830507.926308:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518276608/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830507.926315:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830507.926318:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830507.926321:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518276608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830507.926326:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518276736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830507.927138:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830507.927152:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830507/real 1787830507] req@ffff8b2e880392c0 x1874672518276736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830507.927165:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518276736/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830532 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830507.927173:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830507.927177:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830507.927180:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518276736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830508.244951:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830508.756779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830508.756877:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830508.948770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830508.948770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830509.269262:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830509.780763:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830509.781265:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830509.972799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830509.973200:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830510.292799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830510.804795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830510.805389:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830510.996892:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830510.996950:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830511.316872:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830511.828777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830511.830305:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830512.021283:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830512.021426:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830512.340800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830512.852756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830512.852953:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830513.044785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830513.044785:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830513.044907:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830513.044913:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830513.044917:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830513.044926:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3519 00000100:00080000:3.0:1787830513.044936:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830513.044943:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830513.044986:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518276864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830513.045048:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518276864:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830513.045103:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830513.045132:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830513.045136:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830513.045138:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830513.045142:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3519 00000100:00080000:3.0:1787830513.045146:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830513.045151:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830513.045160:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830513.045180:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518276992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830513.045401:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830513.045405:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830513.045407:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830513.045411:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3519 00000100:00080000:3.0:1787830513.045416:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830513.045420:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830513.045425:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830513.045439:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518277120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830513.045449:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830513.045451:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830513.045453:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830513.045456:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3519 00000100:00080000:3.0:1787830513.045459:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830513.045463:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830513.045467:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830513.045477:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518277248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830513.045485:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830513.045488:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830513.045489:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830513.045493:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3519 00000100:00080000:3.0:1787830513.045496:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830513.045499:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830513.045503:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830513.045511:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518277376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830513.045519:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830513.045522:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830513.045524:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830513.045527:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3519 00000100:00080000:3.0:1787830513.045530:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830513.045532:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830513.045536:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830513.045563:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518277504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830513.045577:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830513.045580:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830513.045582:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830513.045585:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3519 00000100:00080000:3.0:1787830513.045588:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830513.045591:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830513.045595:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830513.045603:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518277632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830513.045696:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830513/real 1787830513] req@ffff8b2e8803a1c0 x1874672518276864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830513.045712:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518276864/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830513.045721:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830513.045725:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830513.045728:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518276864:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830513.045750:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518276992:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830513.045777:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830513.045790:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830513/real 1787830513] req@ffff8b2e88039e00 x1874672518276992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830513.045801:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518276992/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830513.045810:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830513.045812:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830513.045816:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518276992:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830513.045821:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518277120:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830513.045839:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830513.045848:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830513/real 1787830513] req@ffff8b2e8803b840 x1874672518277120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830513.045858:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518277120/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830513.045866:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830513.045869:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830513.045872:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518277120:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830513.045877:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518277248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830513.045905:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830513.045916:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830513/real 1787830513] req@ffff8b2e88039680 x1874672518277248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830513.045927:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518277248/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830513.045934:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830513.045937:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830513.045940:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518277248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830513.045945:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518277376:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830513.045961:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830513.045970:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830513/real 1787830513] req@ffff8b2e880392c0 x1874672518277376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830513.045981:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518277376/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830513.045989:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830513.045992:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830513.045995:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518277376:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830513.046000:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518277504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830513.046014:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830513.046023:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830513/real 1787830513] req@ffff8b2e8803b480 x1874672518277504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830513.046033:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518277504/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830513.046041:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830513.046044:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830513.046047:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518277504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830513.046052:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518277632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830513.046068:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830513.046077:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830513/real 1787830513] req@ffff8b2e8803a580 x1874672518277632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830513.046087:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518277632/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830538 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830513.046095:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830513.046098:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830513.046100:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518277632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830513.364742:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830513.876738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830513.876819:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830514.068782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830514.069860:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830514.388785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830514.901595:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830514.902477:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830515.092785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830515.092787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830515.412865:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830515.924770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830515.924868:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830516.117674:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830516.118337:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830516.436882:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830516.948814:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830516.948856:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830517.140780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830517.140780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830517.460784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830517.972783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830517.972867:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830518.164778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830518.164799:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830518.164805:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830518.164808:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830518.164817:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3524 00000100:00080000:3.0:1787830518.164828:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830518.164835:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830518.164872:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518277760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830518.164944:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518277760:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830518.164964:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000800:00000100:0.0:1787830518.165001:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830518.165032:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830518.165045:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830518.165047:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830518.165052:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3524 00000100:00080000:3.0:1787830518.165055:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830518.165060:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830518.165070:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830518.165082:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518277888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830518.165397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830518/real 1787830518] req@ffff8b2e88039a40 x1874672518277760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830518.165413:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518277760/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830518.165423:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830518.165426:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830518.165430:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518277760:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830518.165452:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518277888:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830518.165475:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830518.165487:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830518/real 1787830518] req@ffff8b2e880383c0 x1874672518277888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830518.165499:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518277888/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830518.165516:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830518.165520:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830518.165522:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518277888:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830518.165609:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830518.165614:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830518.165616:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830518.165620:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3524 00000100:00080000:3.0:1787830518.165624:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830518.165628:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830518.165634:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830518.165701:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518278016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830518.165736:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518278016:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830518.165757:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830518.165825:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830518.165829:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830518.165831:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830518.165834:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3524 00000100:00080000:3.0:1787830518.165838:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830518.165841:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830518.165846:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830518.165857:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518278144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830518.165889:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830518/real 1787830518] req@ffff8b2e88038f00 x1874672518278016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830518.165901:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518278016/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830518.165908:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830518.165911:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830518.165915:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518278016:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830518.165925:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518278144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830518.165942:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830518.165951:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830518/real 1787830518] req@ffff8b2e8803b0c0 x1874672518278144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830518.165961:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518278144/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830518.165968:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830518.165971:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830518.165974:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518278144:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830518.166138:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830518.166142:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830518.166144:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830518.166148:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3524 00000100:00080000:3.0:1787830518.166152:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830518.166156:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830518.166161:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830518.166173:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518278272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830518.166214:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518278272:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830518.166238:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830518.166257:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830518.166260:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830518.166262:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830518.166266:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3524 00000100:00080000:3.0:1787830518.166269:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830518.166273:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830518.166278:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830518.166288:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518278400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830518.166326:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830518/real 1787830518] req@ffff8b2e8803b0c0 x1874672518278272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830518.166339:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518278272/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830518.166347:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830518.166349:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830518.166352:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518278272:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830518.166363:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518278400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830518.166383:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830518.166395:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830518/real 1787830518] req@ffff8b2e88038f00 x1874672518278400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830518.166406:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518278400/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830518.166413:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830518.166416:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830518.166419:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518278400:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830518.166462:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830518.166466:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830518.166468:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830518.166472:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3524 00000100:00080000:3.0:1787830518.166476:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830518.166479:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830518.166484:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830518.166496:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518278528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830518.166735:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518278528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830518.166817:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830518.166830:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830518/real 1787830518] req@ffff8b2e880383c0 x1874672518278528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830518.166848:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518278528/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830543 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830518.166856:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830518.166859:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830518.166862:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518278528:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830518.484905:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830519.001606:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830519.003635:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830519.188845:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830519.188847:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830519.508802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830520.020930:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830520.023122:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830520.213165:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830520.213165:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830520.534781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830521.044782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830521.044987:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830521.236796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830521.236921:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830521.558694:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830522.068768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830522.069338:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830522.260823:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830522.260947:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830522.580781:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830523.092868:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830523.093864:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:3.0:1787830523.284771:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830523.284774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830523.284774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830523.284783:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830523.284788:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830523.284799:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3529 00000100:00080000:3.0:1787830523.284811:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830523.284818:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830523.284856:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518278656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830523.284882:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830523.284886:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830523.284888:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830523.284892:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3529 00000100:00080000:3.0:1787830523.284895:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830523.284898:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830523.284908:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830523.284920:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518278784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830523.284930:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830523.284934:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830523.284936:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830523.284940:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3529 00000100:00080000:3.0:1787830523.284943:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830523.284949:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830523.284950:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518278656:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787830523.284954:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830523.284968:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518278912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830523.284979:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830523.284982:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830523.284984:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830523.284988:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3529 00000100:00080000:3.0:1787830523.284991:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830523.284995:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830523.284999:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830523.285009:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518279040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830523.285018:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830523.285021:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830523.285023:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830523.285027:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3529 00000100:00080000:3.0:1787830523.285030:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830523.285034:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830523.285039:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830523.285050:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518279168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830523.285052:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518278784:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830523.285067:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830523.285071:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830523.285073:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830523.285077:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3529 00000100:00080000:3.0:1787830523.285082:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830523.285086:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830523.285090:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830523.285101:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518279296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830523.285112:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830523.285114:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518278912:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830523.285117:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830523.285121:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830523.285126:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3529 00000100:00080000:3.0:1787830523.285129:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830523.285132:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830523.285138:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830523.285147:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518279040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830523.285151:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518279424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830523.285173:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518279168:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830523.285197:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518279296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830523.285217:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518279424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830523.605078:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830524.116838:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830524.116898:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830524.308799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830524.308856:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830524.628875:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830525.142668:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830525.142814:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830525.332799:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830525.332842:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830525.652980:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830526.164911:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830526.165532:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830526.357028:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830526.357033:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:3.0:1787830526.357050:0:2337:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:3.0:1787830526.357098:0:2337:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787830526.357115:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830526.357148:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830526.357158:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830526.357164:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830526.357171:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830526.357180:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830526.357189:0:2337:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787830526.357381:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830523/real 1787830526] req@ffff8b2e8803b840 x1874672518279296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830526.357403:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518279296/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830526.357426:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830526.357432:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830526.357437:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518279296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830526.357444:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830523/real 1787830526] req@ffff8b2e88039e00 x1874672518279424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830526.357470:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518279424/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830526.357478:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830526.357481:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830526.357485:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518279424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830526.357491:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830523/real 1787830526] req@ffff8b2e8803b480 x1874672518278912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830526.357503:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518278912/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830526.357511:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830526.357514:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830526.357517:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518278912:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830526.357524:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830523/real 1787830526] req@ffff8b2e880392c0 x1874672518279040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830526.357548:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518279040/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830526.357556:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830526.357558:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830526.357562:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518279040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830526.357567:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830523/real 1787830526] req@ffff8b2e88039680 x1874672518279168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830526.357578:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518279168/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830526.357585:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830526.357589:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830526.357592:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518279168:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830526.357598:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830523/real 1787830526] req@ffff8b2e88039a40 x1874672518278656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830526.357610:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518278656/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830526.357618:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830526.357622:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830526.357625:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518278656:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830526.357631:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830523/real 1787830526] req@ffff8b2e8803a580 x1874672518278784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830526.357642:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518278784/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830548 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830526.357669:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830526.357673:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830526.357677:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518278784:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830526.676783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:0.0:1787830527.189005:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830527.189199:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830527.380776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830527.380829:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830527.700783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830528.214761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830528.215151:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830528.404790:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830528.404802:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830528.404815:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830528.404827:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3534 00000100:00080000:3.0:1787830528.404838:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830528.404848:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830528.404852:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830528.404886:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518279552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830528.404963:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518279552:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830528.405031:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830528.405063:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830528.405067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830528.405070:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830528.405075:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3534 00000100:00080000:3.0:1787830528.405080:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830528.405085:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830528.405096:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830528.405112:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830528.405113:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518279680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830528.407967:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830528/real 1787830528] req@ffff8b2e8803a1c0 x1874672518279552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830528.409445:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830528.409456:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830528.409460:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830528.409469:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3534 00000100:00100000:0.0:1787830528.409593:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518279552/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830528.409603:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830528.409606:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830528.409609:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518279552:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830528.409634:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518279680:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830528.410127:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830528.410148:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830528/real 1787830528] req@ffff8b2e8803a580 x1874672518279680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830528.410253:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830528.410261:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830528.410272:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830528.410296:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518279808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830528.410308:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830528.410311:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830528.410313:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830528.410317:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3534 00000100:00080000:3.0:1787830528.410320:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830528.410323:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830528.410327:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830528.410335:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518279936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830528.410343:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830528.410346:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830528.410347:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830528.410351:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3534 00000100:00080000:3.0:1787830528.410353:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830528.410356:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830528.410361:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830528.410370:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518280064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830528.410377:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830528.410380:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830528.410382:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830528.410385:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3534 00000100:00080000:3.0:1787830528.410387:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830528.410391:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830528.410395:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830528.410403:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518280192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830528.410412:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830528.410415:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830528.410416:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830528.410419:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3534 00000100:00080000:3.0:1787830528.410421:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830528.410424:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830528.410427:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830528.410450:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518280320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830528.410604:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518279680/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830528.410614:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830528.410618:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830528.410621:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518279680:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830528.410637:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518279808:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830528.410913:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830528.410927:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830528/real 1787830528] req@ffff8b2e88039a40 x1874672518279808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830528.410938:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518279808/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830528.410945:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830528.410947:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830528.410950:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518279808:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830528.410956:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518279936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830528.410975:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830528.410986:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830528/real 1787830528] req@ffff8b2e88039680 x1874672518279936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830528.410996:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518279936/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830528.411002:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830528.411005:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830528.411008:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518279936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830528.411013:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518280064:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830528.411031:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830528.411041:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830528/real 1787830528] req@ffff8b2e880392c0 x1874672518280064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830528.411056:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518280064/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830528.411064:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830528.411066:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830528.411069:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518280064:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830528.411074:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518280192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830528.411091:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830528.411102:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830528/real 1787830528] req@ffff8b2e8803b480 x1874672518280192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830528.411113:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518280192/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830528.411120:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830528.411122:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830528.411125:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518280192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830528.411130:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518280320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830528.411147:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830528.411465:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830528/real 1787830528] req@ffff8b2e88039e00 x1874672518280320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830528.411482:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518280320/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830553 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830528.411495:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830528.411499:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830528.411501:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518280320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830528.724785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830529.236946:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830529.237039:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830529.428966:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830529.429927:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830529.748784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830530.260993:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830530.263534:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830530.452805:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830530.452821:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830530.772784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830531.284795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830531.285546:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830531.476775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830531.476829:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830531.796900:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830532.308782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830532.309892:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830532.500810:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830532.502630:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830532.820789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830533.332821:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830533.333039:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830533.524754:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830533.524754:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830533.524757:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830533.524767:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830533.524771:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830533.524781:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3539 00000100:00080000:3.0:1787830533.524792:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830533.524799:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830533.524839:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518280448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830533.524871:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830533.524874:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830533.524876:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830533.524880:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3539 00000100:00080000:3.0:1787830533.524883:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830533.524886:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830533.524896:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830533.524915:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518280576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830533.524928:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830533.524931:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830533.524944:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830533.524953:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3539 00000100:00080000:3.0:1787830533.524957:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830533.524960:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830533.524965:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830533.524976:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518280704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830533.524982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830533.524984:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830533.524985:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830533.524988:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3539 00000100:00080000:3.0:1787830533.524990:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830533.524992:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830533.524994:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830533.524999:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518280832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830533.525004:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830533.525006:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830533.525007:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830533.525009:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3539 00000100:00080000:3.0:1787830533.525011:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830533.525014:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830533.525016:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830533.525022:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518280960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830533.525027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830533.525029:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830533.525030:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830533.525032:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3539 00000100:00080000:3.0:1787830533.525034:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830533.525036:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830533.525038:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830533.525043:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518281088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830533.525048:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518280448:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830533.525050:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830533.525055:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830533.525058:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830533.525062:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3539 00000100:00080000:3.0:1787830533.525063:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830533.525065:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830533.525068:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830533.525075:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518281216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830533.525115:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830533.525171:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830533/real 1787830533] req@ffff8b2e8803b840 x1874672518280448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830533.525187:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518280448/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830533.525196:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830533.525200:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830533.525203:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518280448:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830533.525209:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518280576:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830533.525231:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830533.525243:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830533/real 1787830533] req@ffff8b2e880383c0 x1874672518280576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830533.525256:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518280576/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830533.525269:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830533.525272:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830533.525276:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518280576:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830533.525282:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518280704:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830533.525303:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830533.525314:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830533/real 1787830533] req@ffff8b2e88038f00 x1874672518280704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830533.525325:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518280704/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830533.525334:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830533.525336:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830533.525340:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518280704:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830533.525345:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518280832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830533.525360:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830533.525369:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830533/real 1787830533] req@ffff8b2e8803b0c0 x1874672518280832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830533.525380:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518280832/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830533.525387:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830533.525390:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830533.525393:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518280832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830533.525439:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518280960:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830533.525456:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830533.525467:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830533/real 1787830533] req@ffff8b2e88039e00 x1874672518280960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830533.525478:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518280960/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830533.525487:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830533.525489:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830533.525493:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518280960:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830533.525498:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518281088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830533.525514:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830533.525524:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830533/real 1787830533] req@ffff8b2e8803b480 x1874672518281088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830533.525535:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518281088/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830533.525541:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830533.525544:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830533.525547:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518281088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830533.525552:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518281216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830533.525566:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830533.525574:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830533/real 1787830533] req@ffff8b2e880392c0 x1874672518281216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830533.525582:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518281216/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830558 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830533.525594:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830533.525597:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830533.525600:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518281216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830533.844800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830534.356759:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830534.356947:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830534.548794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830534.549296:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830534.868776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830535.380782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830535.381130:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830535.572832:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830535.572835:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830535.893467:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830536.404769:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830536.405061:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830536.597296:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830536.597296:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830536.916798:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830537.428761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830537.428874:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830537.620807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830537.621731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830537.941403:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830538.452776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830538.452869:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830538.644772:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830538.644784:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830538.644789:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830538.644799:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3545 00000100:00080000:3.0:1787830538.644811:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830538.644818:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830538.644854:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830538.644859:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518281344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830538.645002:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518281344:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830538.645032:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000800:00000100:0.0:1787830538.645058:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830538.645086:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830538.645090:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830538.645092:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830538.645097:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3545 00000100:00080000:3.0:1787830538.645101:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830538.645105:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830538.645115:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830538.645129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518281472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830538.645225:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830538/real 1787830538] req@ffff8b2e88039680 x1874672518281344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830538.645242:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518281344/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830538.645251:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830538.645255:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830538.645259:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518281344:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830538.645281:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518281472:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830538.645304:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830538.645317:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830538/real 1787830538] req@ffff8b2e88039a40 x1874672518281472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830538.645328:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518281472/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830538.645336:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830538.645339:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830538.645343:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518281472:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830538.645432:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830538.645436:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830538.645438:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830538.645442:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3545 00000100:00080000:3.0:1787830538.645446:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830538.645450:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830538.645455:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830538.645468:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518281600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830538.645509:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518281600:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830538.645533:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830538.645554:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830538.645558:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830538.645560:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830538.645564:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3545 00000100:00080000:3.0:1787830538.645568:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830538.645572:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830538.645586:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830538.645599:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518281728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830538.645780:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830538/real 1787830538] req@ffff8b2e8803a580 x1874672518281600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830538.645794:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518281600/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830538.645802:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830538.645805:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830538.645808:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518281600:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830538.645820:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518281728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830538.645842:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830538.645854:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830538/real 1787830538] req@ffff8b2e8803a1c0 x1874672518281728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830538.645866:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518281728/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830538.645874:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830538.645877:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830538.645880:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518281728:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830538.646469:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830538.646474:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830538.646476:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830538.646480:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3545 00000100:00080000:3.0:1787830538.646485:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830538.646489:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830538.646494:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830538.646509:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518281856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830538.646518:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830538.646522:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830538.646523:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830538.646526:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3545 00000100:00080000:3.0:1787830538.646530:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830538.646533:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830538.646538:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830538.646548:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518281984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830538.646562:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830538.646564:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830538.646566:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830538.646569:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3545 00000100:00080000:3.0:1787830538.646572:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830538.646575:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830538.646579:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830538.646589:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518282112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830538.646764:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518281856:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830538.646789:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830538.646802:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830538/real 1787830538] req@ffff8b2e8803a1c0 x1874672518281856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830538.646814:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518281856/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830538.646823:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830538.646826:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830538.646829:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518281856:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830538.646834:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518281984:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830538.646850:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830538.646860:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830538/real 1787830538] req@ffff8b2e8803a580 x1874672518281984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830538.646871:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518281984/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830538.646878:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830538.646881:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830538.646884:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518281984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830538.646889:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518282112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830538.646905:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830538.646914:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830538/real 1787830538] req@ffff8b2e88039a40 x1874672518282112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830538.646924:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518282112/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830563 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830538.646935:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830538.646938:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830538.646941:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518282112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830538.964795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830539.476771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830539.476934:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830539.668804:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830539.668804:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830539.988804:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830540.500775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830540.501014:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830540.694144:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830540.695708:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830541.013170:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830541.525128:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830541.525142:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830541.716775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830541.716873:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830542.036986:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830542.548949:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830542.551256:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830542.740777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830542.740911:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830543.060809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830543.572852:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830543.573037:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830543.764773:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830543.764776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830543.764785:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830543.764790:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830543.764800:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3550 00000100:00080000:3.0:1787830543.764811:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830543.764819:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830543.764859:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518282240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830543.764884:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830543.764887:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830543.764889:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830543.764893:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3550 00000100:00080000:3.0:1787830543.764896:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830543.764900:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830543.764910:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830543.764922:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518282368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830543.764932:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830543.764935:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830543.764937:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830543.764941:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3550 00000100:00080000:3.0:1787830543.764945:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830543.764948:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830543.764952:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830543.764964:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518282496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830543.764983:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830543.764987:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830543.764988:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830543.764992:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3550 00000100:00080000:3.0:1787830543.764995:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830543.764998:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830543.765003:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830543.765013:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518282624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830543.765021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830543.765024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830543.765026:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830543.765030:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3550 00000100:00080000:3.0:1787830543.765033:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830543.765036:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830543.765040:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830543.765049:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518282752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830543.765056:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830543.765059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830543.765061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830543.765064:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3550 00000100:00080000:3.0:1787830543.765066:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830543.765069:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830543.765073:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830543.765082:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518282880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830543.765090:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830543.765092:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830543.765094:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830543.765097:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3550 00000100:00080000:3.0:1787830543.765100:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830543.765103:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830543.765107:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830543.765117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518283008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830543.765290:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518282240:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830543.765308:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000800:00000100:0.0:1787830543.765361:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830543.765672:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830543/real 1787830543] req@ffff8b2e88039680 x1874672518282240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830543.765690:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518282240/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830543.765700:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830543.765704:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830543.765708:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518282240:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830543.765715:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518282368:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830543.765742:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830543.765755:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830543/real 1787830543] req@ffff8b2e880392c0 x1874672518282368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830543.765766:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518282368/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830543.765779:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830543.765782:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830543.765785:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518282368:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830543.765791:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518282496:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830543.765823:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830543.765835:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830543/real 1787830543] req@ffff8b2e8803b480 x1874672518282496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830543.765845:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518282496/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830543.765852:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830543.765855:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830543.765858:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518282496:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830543.765863:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518282624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830543.765879:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830543.765889:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830543/real 1787830543] req@ffff8b2e88039e00 x1874672518282624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830543.765899:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518282624/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830543.765906:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830543.765909:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830543.765912:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518282624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830543.765917:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518282752:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830543.765933:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830543.765942:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830543/real 1787830543] req@ffff8b2e8803b0c0 x1874672518282752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830543.765952:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518282752/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830543.765959:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830543.765962:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830543.765965:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518282752:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830543.765971:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518282880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830543.765989:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830543.765999:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830543/real 1787830543] req@ffff8b2e88038f00 x1874672518282880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830543.766009:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518282880/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830543.766016:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830543.766019:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830543.766022:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518282880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830543.766028:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518283008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830543.766059:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830543.766081:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830543/real 1787830543] req@ffff8b2e880383c0 x1874672518283008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830543.766092:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518283008/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830568 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830543.766099:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830543.766102:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830543.766105:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518283008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830544.084782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830544.596756:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830544.597384:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830544.788832:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830544.791147:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830545.109055:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830545.620778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830545.620868:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830545.812789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830545.812789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830546.132773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830546.645981:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830546.648586:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830546.836815:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830546.837549:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830547.156896:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830547.668794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830547.671410:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830547.861507:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830547.861643:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830548.180787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830548.692787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830548.693343:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830548.884781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830548.884793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:2.0:1787830548.884843:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830548.884849:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830548.884852:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830548.884862:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3555 00000100:00080000:2.0:1787830548.884874:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830548.884881:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830548.884920:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518283136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830548.884946:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830548.884949:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830548.884951:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830548.884955:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3555 00000100:00080000:2.0:1787830548.884958:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830548.884962:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830548.884972:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830548.884985:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518283264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830548.884995:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518283136:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787830548.884999:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830548.885004:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830548.885006:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830548.885010:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3555 00000100:00080000:2.0:1787830548.885014:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830548.885018:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830548.885023:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830548.885034:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518283392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830548.885043:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830548.885046:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830548.885048:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830548.885051:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3555 00000100:00080000:2.0:1787830548.885054:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:0.0:1787830548.885061:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787830548.885061:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830548.885069:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830548.885080:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518283520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830548.885089:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830548.885092:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830548.885094:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830548.885099:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3555 00000100:00080000:2.0:1787830548.885103:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830548.885107:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830548.885110:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830548/real 1787830548] req@ffff8b2e8801de00 x1874672518283136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787830548.885124:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830548.885128:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518283136/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830548.885139:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:2.0:1787830548.885141:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518283648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830548.885145:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830548.885153:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518283136:192.168.203.101@tcp:250:kworker.0 00000100:00080000:2.0:1787830548.885158:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830548.885163:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830548.885165:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830548.885171:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3555 00000100:00080000:2.0:1787830548.885174:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830548.885178:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830548.885181:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518283264:192.168.203.101@tcp:38:kworker.0 00000008:00100000:2.0:1787830548.885185:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830548.885198:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518283776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830548.885207:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787830548.885212:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:2.0:1787830548.885214:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830548.885217:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830548.885228:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830548/real 1787830548] req@ffff8b2e8801cb40 x1874672518283264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830548.885231:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3555 00000100:00080000:2.0:1787830548.885238:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830548.885245:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830548.885249:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518283264/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:2.0:1787830548.885252:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830548.885267:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518283904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830548.885557:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830548.885568:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830548.885572:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518283264:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830548.885578:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518283392:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830548.885603:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830548.885616:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830548/real 1787830548] req@ffff8b2e8801c3c0 x1874672518283392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830548.885638:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518283392/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830548.885670:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830548.885682:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830548.885686:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518283392:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830548.885692:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518283520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830548.885713:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830548.885725:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830548/real 1787830548] req@ffff8b2e8801d680 x1874672518283520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830548.885736:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518283520/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830548.885744:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830548.885747:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830548.885751:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518283520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830548.885756:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518283648:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830548.885773:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830548.885783:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830548/real 1787830548] req@ffff8b2e8801f0c0 x1874672518283648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830548.885794:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518283648/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830548.885802:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830548.885805:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830548.885808:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518283648:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830548.885826:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518283776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830548.885856:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830548.885869:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830548/real 1787830548] req@ffff8b2e8801c780 x1874672518283776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830548.885881:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518283776/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830548.885889:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830548.885892:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830548.885895:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518283776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830548.885900:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518283904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830548.885919:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830548.885930:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830548/real 1787830548] req@ffff8b2e8801ed00 x1874672518283904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830548.885942:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518283904/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830573 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830548.885949:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830548.885952:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830548.885955:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518283904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830549.204794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830549.716866:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830549.716929:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830549.908926:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830549.909332:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830550.228792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830550.740792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830550.741105:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830550.932764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830550.932775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830551.252807:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830551.764798:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830551.765326:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830551.956778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830551.956778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830552.276824:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830552.788850:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830552.788980:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830552.980817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830552.980818:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830553.300789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830553.812829:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830553.813120:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830554.004772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830554.004790:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830554.004797:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830554.004801:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830554.004810:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3560 00000100:00080000:0.0:1787830554.004820:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830554.004827:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830554.004866:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518284032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830554.004893:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830554.004897:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830554.004899:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830554.004903:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3560 00000100:00080000:0.0:1787830554.004906:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830554.004910:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830554.004919:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830554.004933:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518284160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830554.004943:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830554.004947:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830554.004949:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830554.004953:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3560 00000100:00080000:0.0:1787830554.004958:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830554.004962:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830554.004967:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830554.004979:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518284288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830554.004983:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518284032:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787830554.004997:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830554.005000:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830554.005002:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830554.005007:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3560 00000100:00080000:0.0:1787830554.005011:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830554.005017:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830554.005020:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000008:00100000:0.0:1787830554.005024:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787830554.005055:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830554.005100:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518284416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830554.005105:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830554/real 1787830554] req@ffff8b2e880403c0 x1874672518284032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830554.005118:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830554.005124:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830554.005126:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830554.005131:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518284032/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830554.005132:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3560 00000100:00080000:0.0:1787830554.005139:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830554.005146:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787830554.005148:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:0.0:1787830554.005152:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787830554.005154:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830554.005160:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518284032:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830554.005165:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518284544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830554.005172:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518284160:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787830554.005180:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830554.005184:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830554.005185:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830554.005190:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3560 00000100:00080000:0.0:1787830554.005194:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830554.005198:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830554.005203:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787830554.005203:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830554.005217:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518284672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830554.005220:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830554/real 1787830554] req@ffff8b2e88040000 x1874672518284160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830554.005239:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830554.005244:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787830554.005245:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518284160/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830554.005247:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830554.005266:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3560 00000100:00080000:0.0:1787830554.005271:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830554.005275:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830554.005280:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830554.005289:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518284800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830554.005711:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830554.005716:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830554.005719:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518284160:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830554.005745:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518284288:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830554.005769:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830554.005781:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830554/real 1787830554] req@ffff8b2e880430c0 x1874672518284288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830554.005793:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518284288/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830554.005801:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830554.005804:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830554.005807:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518284288:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830554.005813:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518284416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830554.005831:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830554.005843:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830554/real 1787830554] req@ffff8b2e88043480 x1874672518284416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830554.005854:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518284416/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830554.005861:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830554.005864:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830554.005868:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518284416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830554.005874:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518284544:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830554.005893:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830554.005904:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830554/real 1787830554] req@ffff8b2e88042940 x1874672518284544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830554.005929:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518284544/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830554.005938:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830554.005942:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830554.005945:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518284544:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830554.005950:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518284672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830554.005969:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830554.005981:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830554/real 1787830554] req@ffff8b2e88040f00 x1874672518284672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830554.005992:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518284672/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830554.006000:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830554.006003:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830554.006006:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518284672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830554.006011:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518284800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830554.006029:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830554.006041:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830554/real 1787830554] req@ffff8b2e88041a40 x1874672518284800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830554.006051:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518284800/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830579 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830554.006059:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830554.006061:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830554.006064:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518284800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830554.325187:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830554.836966:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830554.838891:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830555.029008:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830555.029528:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830555.349212:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830555.860813:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830555.860917:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830556.052856:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830556.054986:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830556.372785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830556.884736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830556.886314:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830557.076778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830557.076884:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830557.397257:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830557.909294:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830557.909667:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830558.100784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830558.100791:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830558.420917:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830558.932795:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830558.932953:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:0.0:1787830559.124759:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830559.124763:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830559.124770:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830559.124775:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830559.124785:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3565 00000100:00080000:0.0:1787830559.124796:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830559.124805:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830559.124821:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830559.124850:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518284928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830559.124879:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830559.124882:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830559.124884:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830559.124888:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3565 00000100:00080000:0.0:1787830559.124892:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830559.124896:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830559.124905:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830559.124918:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518285056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830559.124928:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830559.124930:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830559.124932:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830559.124935:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3565 00000100:00080000:0.0:1787830559.124939:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830559.124942:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830559.124957:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830559.124967:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518285184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830559.124973:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830559.124975:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830559.124976:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830559.124978:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3565 00000100:00080000:0.0:1787830559.124981:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830559.124983:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830559.124985:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830559.124990:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518285312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830559.124995:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830559.124997:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830559.124998:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830559.125000:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3565 00000100:00080000:0.0:1787830559.125002:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830559.125003:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830559.125005:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830559.125010:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518285440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830559.125014:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830559.125016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830559.125017:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830559.125019:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3565 00000100:00080000:0.0:1787830559.125021:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830559.125023:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830559.125025:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830559.125029:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518285568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830559.125034:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830559.125035:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830559.125036:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830559.125038:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3565 00000100:00080000:0.0:1787830559.125039:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830559.125041:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830559.125043:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830559.125047:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518285696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830559.125262:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518284928:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830559.125321:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830559.125364:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830559/real 1787830559] req@ffff8b2e880421c0 x1874672518284928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830559.125428:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518284928/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830559.125439:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830559.125443:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830559.125447:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518284928:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830559.125455:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518285056:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830559.125478:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830559.125557:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830559/real 1787830559] req@ffff8b2e88040b40 x1874672518285056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830559.125571:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518285056/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830559.125579:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830559.125582:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830559.125585:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518285056:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830559.125591:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518285184:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830559.125612:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830559.125623:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830559/real 1787830559] req@ffff8b2e880412c0 x1874672518285184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830559.125634:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518285184/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830559.125641:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830559.125664:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830559.125668:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518285184:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830559.125674:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518285312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830559.125693:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830559.126201:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830559/real 1787830559] req@ffff8b2e88043840 x1874672518285312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830559.126224:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518285312/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830559.126234:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830559.126238:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830559.126241:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518285312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830559.126247:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518285440:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830559.126273:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830559.126285:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830559/real 1787830559] req@ffff8b2e88040780 x1874672518285440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830559.126296:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518285440/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830559.126303:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830559.126306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830559.126309:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518285440:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830559.126316:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518285568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830559.126334:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830559.126344:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830559/real 1787830559] req@ffff8b2e88042d00 x1874672518285568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830559.126354:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518285568/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830559.126360:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830559.126363:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830559.126366:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518285568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830559.126371:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518285696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830559.126398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830559.126408:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830559/real 1787830559] req@ffff8b2e88042580 x1874672518285696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830559.126417:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518285696/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830584 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830559.126424:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830559.126427:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830559.126430:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518285696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830559.444834:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830559.956772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830559.956856:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830560.148787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830560.149145:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830560.468816:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830560.980829:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830560.981092:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830561.172778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830561.172778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830561.492918:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830562.004779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830562.004881:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830562.196789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830562.197089:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830562.516799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830563.028849:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830563.028908:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830563.220790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830563.221003:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830563.540790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830564.052772:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830564.052844:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830564.244797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830564.245279:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830564.245286:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830564.245290:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830564.245299:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3570 00000100:00100000:2.0:1787830564.245304:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830564.245310:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830564.245319:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830564.245355:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518285824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830564.245385:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830564.245391:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830564.245392:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830564.245396:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3570 00000100:00080000:3.0:1787830564.245400:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830564.245403:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830564.245413:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830564.245426:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518285952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830564.245435:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830564.245438:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830564.245440:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830564.245443:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3570 00000100:00080000:3.0:1787830564.245447:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830564.245450:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830564.245455:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830564.245465:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518286080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830564.245473:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830564.245476:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830564.245477:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830564.245480:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3570 00000100:00080000:3.0:1787830564.245483:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830564.245486:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830564.245499:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830564.245510:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518286208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830564.245518:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830564.245521:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830564.245523:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830564.245527:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3570 00000100:00080000:3.0:1787830564.245530:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830564.245533:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830564.245686:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518285824:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830564.245731:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830564.245773:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830564/real 1787830564] req@ffff8b2e8803b840 x1874672518285824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830564.245800:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518285824/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830564.245816:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830564.245821:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830564.245825:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518285824:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830564.245831:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518285952:192.168.203.101@tcp:38:kworker.0 00000008:00100000:3.0:1787830564.245860:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830564.245866:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830564.245876:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518286336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830564.245888:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830564.245891:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830564.245893:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830564.245897:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3570 00000100:00100000:0.0:1787830564.245900:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830564/real 1787830564] req@ffff8b2e880383c0 x1874672518285952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830564.245903:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830564.245910:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830564.245918:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830564.245923:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518285952/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830564.245930:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518286464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830564.245937:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830564.245944:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787830564.245945:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830564.245950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787830564.245951:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518285952:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830564.245953:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830564.245960:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3570 00000100:00080000:3.0:1787830564.245962:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830564.245965:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518286080:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830564.245968:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830564.245974:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830564.245984:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518286592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830564.245996:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830564.246011:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830564/real 1787830564] req@ffff8b2e88038f00 x1874672518286080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830564.246024:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518286080/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830564.246032:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830564.246035:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830564.246038:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518286080:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830564.246045:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518286208:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830564.246064:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830564.246074:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830564/real 1787830564] req@ffff8b2e8803b0c0 x1874672518286208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830564.246086:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518286208/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830564.246095:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830564.246097:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830564.246100:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518286208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830564.246137:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518286336:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830564.246157:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830564.246168:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830564/real 1787830564] req@ffff8b2e88039e00 x1874672518286336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830564.246179:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518286336/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830564.246202:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830564.246205:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830564.246208:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518286336:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830564.246214:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518286464:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830564.246233:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830564.246244:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830564/real 1787830564] req@ffff8b2e8803b480 x1874672518286464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830564.246273:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518286464/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830564.246281:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830564.246284:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830564.246287:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518286464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830564.246293:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518286592:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830564.246311:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830564.246321:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830564/real 1787830564] req@ffff8b2e880392c0 x1874672518286592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830564.246332:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518286592/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830589 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830564.246340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830564.246343:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830564.246346:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518286592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830564.564780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830565.076828:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830565.076942:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830565.269012:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830565.269143:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830565.588879:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830566.101185:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830566.101387:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830566.292778:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830566.292790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830566.612909:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830567.124808:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830567.125204:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830567.316790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830567.317238:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830567.638528:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830568.149021:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830568.149776:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830568.340794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830568.342250:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830568.660790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830569.172903:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830569.173352:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830569.364900:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830569.365209:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830569.365215:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830569.365219:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830569.365226:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830569.365230:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3575 00000100:00080000:1.0:1787830569.365241:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830569.365249:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830569.365286:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518286720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830569.365309:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830569.365313:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830569.365315:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830569.365319:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3575 00000100:00080000:1.0:1787830569.365323:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830569.365327:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830569.365337:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830569.365341:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518286720:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830569.365354:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518286848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830569.365365:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830569.365368:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830569.365371:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830569.365375:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3575 00000100:00080000:1.0:1787830569.365389:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830569.365393:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:0.0:1787830569.365399:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787830569.365399:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830569.365414:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518286976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830569.365423:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830569.365426:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830569.365428:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830569.365431:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3575 00000100:00080000:1.0:1787830569.365435:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830569.365439:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830569.365444:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830569/real 1787830569] req@ffff8b2e88076580 x1874672518286720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787830569.365445:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830569.365460:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518287104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830569.365473:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830569.365476:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830569.365478:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830569.365483:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518286720/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830569.365485:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3575 00000100:00080000:1.0:1787830569.365492:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830569.365498:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787830569.365501:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787830569.365503:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787830569.365509:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830569.365514:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518286720:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830569.365518:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518287232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830569.365529:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830569.365532:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830569.365533:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830569.365537:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3575 00000100:00100000:0.0:1787830569.365541:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518286848:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830569.365543:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830569.365550:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830569.365556:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830569.365567:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518287360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830569.365573:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830569.365583:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830569.365587:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830569.365589:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830569.365591:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830569/real 1787830569] req@ffff8b2e88075e00 x1874672518286848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830569.365594:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3575 00000100:00080000:1.0:1787830569.365601:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830569.365608:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830569.365612:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518286848/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787830569.365613:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787830569.365627:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787830569.365629:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518287488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830569.365634:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830569.365642:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518286848:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830569.365682:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518286976:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830569.365702:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830569.365713:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830569/real 1787830569] req@ffff8b2e88074b40 x1874672518286976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830569.365724:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518286976/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830569.365731:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830569.365734:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830569.365736:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518286976:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830569.365743:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518287104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830569.365759:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830569.365768:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830569/real 1787830569] req@ffff8b2e88074000 x1874672518287104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830569.365777:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518287104/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830569.365784:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830569.365787:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830569.365790:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518287104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830569.365795:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518287232:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830569.365812:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830569.365822:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830569/real 1787830569] req@ffff8b2e880761c0 x1874672518287232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830569.365833:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518287232/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830569.365841:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830569.365844:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830569.365847:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518287232:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830569.365862:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518287360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830569.365881:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830569.365892:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830569/real 1787830569] req@ffff8b2e880770c0 x1874672518287360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830569.365902:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518287360/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830569.365909:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830569.365912:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830569.365915:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518287360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830569.365920:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518287488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830569.365936:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830569.365944:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830569/real 1787830569] req@ffff8b2e88077840 x1874672518287488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830569.365954:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518287488/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830594 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830569.365961:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830569.365963:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830569.365966:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518287488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830569.684793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830570.196781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830570.198199:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830570.388785:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830570.391257:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830570.709163:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830571.220793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830571.221969:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830571.412855:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830571.413011:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830571.732867:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830572.244779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830572.245167:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830572.436796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830572.437194:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830572.756772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830573.268871:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830573.269082:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830573.460787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830573.460878:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830573.781312:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830574.292845:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830574.294394:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830574.484771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830574.485195:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830574.485201:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830574.485205:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830574.485214:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3580 00000100:00080000:0.0:1787830574.485224:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830574.485230:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830574.485269:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518287616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830574.485297:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830574.485300:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830574.485302:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830574.485306:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3580 00000100:00080000:0.0:1787830574.485309:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830574.485312:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830574.485322:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830574.485345:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518287744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830574.485355:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830574.485358:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830574.485360:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830574.485363:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3580 00000100:00080000:0.0:1787830574.485366:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830574.485370:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830574.485375:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830574.485384:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518287872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830574.485391:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830574.485394:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830574.485395:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830574.485397:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3580 00000100:00080000:0.0:1787830574.485399:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830574.485401:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830574.485404:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830574.485410:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518288000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830574.485415:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830574.485417:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830574.485418:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830574.485420:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3580 00000100:00080000:0.0:1787830574.485421:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830574.485423:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830574.485425:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830574.485429:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518288128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830574.485434:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830574.485435:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830574.485436:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830574.485438:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3580 00000100:00080000:0.0:1787830574.485439:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830574.485441:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830574.485444:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830574.485449:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518288256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830574.485453:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830574.485455:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830574.485456:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830574.485457:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3580 00000100:00080000:0.0:1787830574.485459:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830574.485461:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830574.485463:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830574.485467:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518288384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830574.485531:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518287616:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830574.485578:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830574.485613:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830574/real 1787830574] req@ffff8b2e88041e00 x1874672518287616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830574.485623:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518287616/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830574.485629:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830574.485632:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830574.485634:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518287616:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830574.485638:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518287744:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830574.485667:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830574.485676:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830574/real 1787830574] req@ffff8b2e88042580 x1874672518287744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830574.485683:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518287744/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830574.485688:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830574.485690:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830574.485692:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518287744:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830574.485697:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518287872:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830574.485715:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830574.485726:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830574/real 1787830574] req@ffff8b2e88042d00 x1874672518287872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830574.485736:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518287872/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830574.485743:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830574.485745:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830574.485748:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518287872:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830574.485753:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518288000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830574.485771:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830574.485782:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830574/real 1787830574] req@ffff8b2e88040780 x1874672518288000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830574.485799:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518288000/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830574.485805:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830574.485808:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830574.485810:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518288000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830574.485816:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518288128:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830574.485834:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830574.485845:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830574/real 1787830574] req@ffff8b2e88043840 x1874672518288128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830574.485855:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518288128/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830574.485862:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830574.485865:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830574.485868:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518288128:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830574.485873:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518288256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830574.485891:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830574.485901:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830574/real 1787830574] req@ffff8b2e880412c0 x1874672518288256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830574.485912:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518288256/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830574.485923:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830574.485926:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830574.485929:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518288256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830574.485949:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518288384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830574.485972:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830574.485983:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830574/real 1787830574] req@ffff8b2e88040b40 x1874672518288384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830574.485993:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518288384/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830599 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830574.486001:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830574.486005:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830574.486009:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518288384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830574.486027:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830574.804729:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830575.316812:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830575.317126:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830575.508792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830575.508792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830575.828774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830576.340776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830576.340953:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830576.532798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830576.533305:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830576.852990:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830577.364820:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830577.365270:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830577.556792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830577.556792:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830577.877066:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830578.388797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830578.388906:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830578.580777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830578.580777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830578.900784:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830579.412778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830579.412877:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830579.604806:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830579.604819:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830579.604823:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830579.604825:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830579.604836:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3586 00000100:00080000:3.0:1787830579.604847:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00100000:2.0:1787830579.604854:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830579.604855:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830579.604897:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518288512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830579.604926:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830579.604929:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830579.604931:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830579.604934:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3586 00000100:00080000:3.0:1787830579.604938:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830579.604941:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830579.604951:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830579.604961:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518288640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830579.604970:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830579.604973:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830579.604975:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830579.604977:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3586 00000100:00080000:3.0:1787830579.604980:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830579.604983:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830579.604987:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830579.604995:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518288768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830579.605003:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830579.605018:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830579.605021:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:0.0:1787830579.605022:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518288512:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830579.605026:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3586 00000100:00080000:3.0:1787830579.605032:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830579.605035:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830579.605040:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830579.605052:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518288896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830579.605061:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830579.605064:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830579.605066:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830579.605069:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3586 00000100:00080000:3.0:1787830579.605074:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830579.605091:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830579.605096:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830579.605098:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830579.605117:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518289024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830579.605143:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830579/real 1787830579] req@ffff8b2e88039680 x1874672518288512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830579.605159:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518288512/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830579.605168:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830579.605172:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830579.605175:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518288512:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830579.605183:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830579.605184:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518288640:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830579.605187:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830579.605190:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830579.605196:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3586 00000100:00080000:3.0:1787830579.605199:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830579.605202:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830579.605207:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830579.605213:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830579.605218:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518289152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830579.605228:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830579/real 1787830579] req@ffff8b2e88039a40 x1874672518288640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830579.605231:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830579.605240:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830579.605243:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830579.605250:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3586 00000100:00100000:0.0:1787830579.605252:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518288640/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830579.605257:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830579.605265:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:0.0:1787830579.605266:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787830579.605271:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787830579.605273:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830579.605279:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518288640:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830579.605286:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518289280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830579.605289:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518288768:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830579.605318:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830579.605331:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830579/real 1787830579] req@ffff8b2e8803a580 x1874672518288768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830579.605340:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518288768/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830579.605347:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830579.605350:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830579.605352:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518288768:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830579.605375:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518288896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830579.605393:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830579.605402:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830579/real 1787830579] req@ffff8b2e8803a1c0 x1874672518288896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830579.605420:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518288896/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830579.605428:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830579.605431:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830579.605433:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518288896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830579.605439:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518289024:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830579.605456:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830579.605467:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830579/real 1787830579] req@ffff8b2e880392c0 x1874672518289024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830579.605478:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518289024/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830579.605485:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830579.605488:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830579.605491:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518289024:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830579.605495:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518289152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830579.605510:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830579.605517:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830579/real 1787830579] req@ffff8b2e8803b480 x1874672518289152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830579.605526:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518289152/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830579.605537:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830579.605540:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830579.605542:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518289152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830579.605547:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518289280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830579.605559:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830579.605567:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830579/real 1787830579] req@ffff8b2e88039e00 x1874672518289280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830579.605575:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518289280/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830604 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830579.605582:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830579.605584:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830579.605587:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518289280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830579.924780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830580.436799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830580.437014:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830580.628792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830580.628908:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830580.949207:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830581.460796:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830581.462143:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830581.652773:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830581.652773:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830581.973606:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830582.484838:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830582.484864:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830582.676789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830582.676797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830582.996773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830583.508773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830583.509153:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830583.700787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830583.703612:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830584.020844:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830584.532989:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830584.533327:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830584.724771:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830584.724783:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830584.724787:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830584.724796:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3591 00000100:00080000:3.0:1787830584.724807:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830584.724813:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830584.724835:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830584.724853:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518289408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830584.724935:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518289408:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830584.724972:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000800:00000100:0.0:1787830584.725003:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830584.725035:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830584.725047:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830584.725049:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830584.725054:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3591 00000100:00080000:3.0:1787830584.725062:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830584.725067:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830584.725078:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830584.725091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518289536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830584.725184:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830584/real 1787830584] req@ffff8b2e8803b0c0 x1874672518289408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830584.725200:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518289408/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830584.725209:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830584.725212:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830584.725216:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518289408:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830584.725236:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518289536:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830584.725258:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830584.725270:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830584/real 1787830584] req@ffff8b2e88038f00 x1874672518289536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830584.725283:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518289536/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830584.725292:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830584.725295:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830584.725298:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518289536:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830584.725444:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830584.725449:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830584.725450:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830584.725454:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3591 00000100:00080000:3.0:1787830584.725459:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830584.725463:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830584.725468:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830584.725480:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518289664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830584.725531:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518289664:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830584.725554:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830584.725576:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830584.725580:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830584.725581:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830584.725586:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3591 00000100:00080000:3.0:1787830584.725590:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830584.725594:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830584.725599:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830584.725611:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518289792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830584.725777:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830584/real 1787830584] req@ffff8b2e880383c0 x1874672518289664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830584.725791:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518289664/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830584.725798:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830584.725801:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830584.725804:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518289664:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830584.725816:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518289792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830584.725836:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830584.725846:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830584/real 1787830584] req@ffff8b2e8803b840 x1874672518289792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830584.725857:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518289792/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830584.725865:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830584.725868:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830584.725871:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518289792:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830584.725955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830584.725958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830584.725959:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830584.725962:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3591 00000100:00080000:3.0:1787830584.725965:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830584.725968:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830584.725972:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830584.725983:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518289920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830584.725995:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830584.725998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830584.726000:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830584.726007:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3591 00000100:00080000:3.0:1787830584.726011:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830584.726014:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830584.726018:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830584.726026:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518289920:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830584.726027:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518290048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830584.726036:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830584.726051:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830584.726052:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000800:00000100:0.0:1787830584.726078:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830584.726091:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830584/real 1787830584] req@ffff8b2e8803b840 x1874672518289920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830584.726102:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518289920/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830584.726109:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830584.726112:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830584.726115:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518289920:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830584.726125:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518290048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830584.726143:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830584.726144:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3591 00000100:00080000:3.0:1787830584.726147:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830584.726151:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830584.726153:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830584/real 1787830584] req@ffff8b2e880383c0 x1874672518290048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787830584.726156:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830584.726164:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518290048/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830584.726166:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518290176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830584.726178:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830584.726181:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830584.726183:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518290048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830584.726193:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518290176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830584.726211:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830584.726221:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830584/real 1787830584] req@ffff8b2e88038f00 x1874672518290176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830584.726231:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518290176/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830609 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830584.726238:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830584.726241:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830584.726244:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518290176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830585.044933:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830585.556793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830585.556824:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830585.748806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830585.749006:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830586.068788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830586.580774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830586.580828:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830586.772983:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830586.772983:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830587.093007:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830587.604736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830587.604857:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830587.796776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830587.796776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830588.116788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830588.628919:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830588.628937:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830588.820939:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830588.821137:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830589.140779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830589.652823:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830589.652917:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830589.844775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830589.844775:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830589.844790:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830589.844796:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830589.844799:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830589.844806:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3596 00000100:00080000:0.0:1787830589.844816:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830589.844823:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830589.844867:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518290304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830589.844892:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830589.844906:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830589.844908:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830589.844913:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3596 00000100:00080000:0.0:1787830589.844917:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830589.844921:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830589.844930:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830589.844930:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518290304:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830589.844946:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518290432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830589.844958:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830589.844962:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830589.844963:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830589.844967:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3596 00000100:00080000:0.0:1787830589.844971:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830589.844975:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830589.844979:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830589.844992:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518290560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830589.845000:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830589.845016:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830589.845018:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830589.845021:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3596 00000100:00080000:0.0:1787830589.845025:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830589.845029:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830589.845035:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830589.845047:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518290688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830589.845048:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518290432:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787830589.845070:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830589.845073:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830589.845074:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830589.845077:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3596 00000100:00080000:0.0:1787830589.845080:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830589.845083:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830589.845088:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830589.845088:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518290560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830589.845098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518290816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830589.845105:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830589.845107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830589.845108:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830589.845114:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3596 00000100:00080000:0.0:1787830589.845117:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830589.845119:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830589.845123:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830589.845124:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518290688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830589.845132:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518290944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830589.845140:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830589.845142:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830589.845143:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830589.845146:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3596 00000100:00080000:0.0:1787830589.845149:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830589.845154:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830589.845155:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518290816:192.168.203.101@tcp:38:kworker.0 00000008:00100000:0.0:1787830589.845158:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830589.845168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518291072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830589.845185:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518290944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830589.845197:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518291072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830590.164893:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830590.676886:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830590.677162:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830590.868782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830590.868841:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830591.188789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830591.700947:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830591.701180:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830591.892771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830591.892951:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830592.212793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830592.724852:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830592.724926:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830592.916779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830592.916926:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:3.0:1787830592.917025:0:2334:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:3.0:1787830592.917069:0:2334:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787830592.917085:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830592.917121:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830592.917126:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830592.917136:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830592.917144:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830592.917150:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830592.917156:0:2334:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:0.0:1787830592.917226:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830589/real 1787830592] req@ffff8b2e880430c0 x1874672518290944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830592.917249:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518290944/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830592.917260:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830592.917264:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830592.917268:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518290944:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830592.917275:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830589/real 1787830592] req@ffff8b2e88040000 x1874672518291072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830592.917286:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518291072/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830592.917299:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830592.917302:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830592.917305:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518291072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830592.917311:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830589/real 1787830592] req@ffff8b2e88042940 x1874672518290688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830592.917323:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518290688/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830592.917331:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830592.917334:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830592.917337:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518290688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830592.917343:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830589/real 1787830592] req@ffff8b2e88043480 x1874672518290816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830592.917354:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518290816/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830592.917362:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830592.917365:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830592.917368:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518290816:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830592.917374:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830589/real 1787830592] req@ffff8b2e88041a40 x1874672518290432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830592.917385:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518290432/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830592.917392:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830592.917395:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830592.917397:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518290432:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830592.917402:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830589/real 1787830592] req@ffff8b2e88040f00 x1874672518290560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830592.917413:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518290560/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830592.917420:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830592.917423:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830592.917426:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518290560:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830592.917431:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830589/real 1787830592] req@ffff8b2e880421c0 x1874672518290304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830592.917443:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518290304/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830614 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830592.917450:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830592.917454:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830592.917457:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518290304:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830593.237185:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830593.748829:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830593.749027:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830593.940839:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830593.943117:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830594.261059:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830594.772793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830594.774160:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830594.964807:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830594.964837:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830594.964964:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830594.964979:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830594.964983:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830594.964992:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3601 00000100:00080000:3.0:1787830594.965001:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830594.965008:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830594.965041:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518291200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830594.965097:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518291200:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830594.965151:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830594.965181:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830594.965185:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830594.965187:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830594.965191:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3601 00000100:00080000:3.0:1787830594.965195:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830594.965200:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830594.965208:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830594.965222:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518291328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830594.965274:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830594/real 1787830594] req@ffff8b2e8803b0c0 x1874672518291200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830594.965290:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518291200/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830594.965299:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830594.965303:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830594.965306:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518291200:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830594.965327:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518291328:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830594.965348:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830594.965361:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830594/real 1787830594] req@ffff8b2e88039e00 x1874672518291328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830594.965373:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518291328/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830594.965380:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830594.965383:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830594.965386:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518291328:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830594.965431:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830594.965435:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830594.965436:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830594.965440:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3601 00000100:00080000:3.0:1787830594.965444:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830594.965448:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830594.965454:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830594.965466:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518291456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830594.965476:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830594.965478:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830594.965480:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830594.965483:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3601 00000100:00080000:3.0:1787830594.965486:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830594.965489:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830594.965493:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830594.965502:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518291584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830594.965510:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830594.965514:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830594.965515:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830594.965520:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3601 00000100:00080000:3.0:1787830594.965523:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830594.965526:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830594.965530:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830594.965539:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518291712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830594.965547:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830594.965549:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830594.965551:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830594.965554:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3601 00000100:00080000:3.0:1787830594.965557:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830594.965560:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830594.965563:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830594.965572:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518291840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830594.965579:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830594.965581:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830594.965583:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830594.965585:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3601 00000100:00080000:3.0:1787830594.965588:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830594.965590:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830594.965594:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830594.965609:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518291968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830594.966186:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518291456:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830594.966221:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830594.966238:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830594/real 1787830594] req@ffff8b2e8803b480 x1874672518291456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830594.966250:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518291456/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830594.966258:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830594.966262:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830594.966265:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518291456:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830594.966271:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518291584:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830594.966290:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830594.966300:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830594/real 1787830594] req@ffff8b2e880392c0 x1874672518291584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830594.966312:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518291584/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830594.966320:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830594.966322:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830594.966325:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518291584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830594.966331:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518291712:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830594.966348:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830594.966358:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830594/real 1787830594] req@ffff8b2e8803a1c0 x1874672518291712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830594.966369:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518291712/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830594.966375:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830594.966378:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830594.966381:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518291712:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830594.966387:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518291840:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830594.966405:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830594.966415:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830594/real 1787830594] req@ffff8b2e8803a580 x1874672518291840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830594.966426:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518291840/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830594.966439:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830594.966442:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830594.966444:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518291840:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830594.966450:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518291968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830594.966468:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830594.966477:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830594/real 1787830594] req@ffff8b2e88039a40 x1874672518291968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830594.966488:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518291968/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830619 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830594.966495:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830594.966499:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830594.966501:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518291968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830595.286626:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830595.796952:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830595.797044:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830595.988781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830595.988900:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830596.310090:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830596.821023:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830596.821855:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830597.012829:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830597.013267:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830597.333261:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830597.844863:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830597.845003:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830598.036804:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830598.036861:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830598.357805:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830598.868780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830598.869933:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830599.060771:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830599.061758:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830599.383093:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830599.892753:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830599.892915:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830600.084774:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830600.084777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830600.084784:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830600.084787:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830600.084796:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3606 00000100:00080000:3.0:1787830600.084805:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830600.084812:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830600.084853:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518292096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830600.084885:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830600.084889:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830600.084891:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830600.084895:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3606 00000100:00080000:3.0:1787830600.084899:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830600.084903:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830600.084913:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830600.084925:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518292224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830600.084950:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830600.084954:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830600.084955:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830600.084959:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3606 00000100:00080000:3.0:1787830600.084962:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830600.084966:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830600.084971:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830600.084982:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830600.084984:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518292352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830600.084998:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830600.085002:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830600.085003:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830600.085007:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3606 00000100:00080000:3.0:1787830600.085010:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830600.085013:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830600.085017:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830600.085027:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518292480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830600.085035:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830600.085039:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830600.085041:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830600.085044:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3606 00000100:00080000:3.0:1787830600.085047:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830600.085051:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830600.085055:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830600.085065:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518292608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830600.085072:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830600.085075:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830600.085077:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830600.085080:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3606 00000100:00080000:3.0:1787830600.085083:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830600.085086:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830600.085089:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830600.085098:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518292736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830600.085106:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830600.085109:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830600.085111:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830600.085114:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3606 00000100:00080000:3.0:1787830600.085116:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830600.085119:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830600.085123:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830600.085131:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518292864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830600.085197:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518292096:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830600.085258:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830600.085294:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830600/real 1787830600] req@ffff8b2e88039680 x1874672518292096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830600.085309:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518292096/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830600.085329:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830600.085334:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830600.085391:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518292096:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830600.085401:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518292224:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830600.085427:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830600.085441:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830600/real 1787830600] req@ffff8b2e88039a40 x1874672518292224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830600.085453:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518292224/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830600.085460:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830600.085463:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830600.085555:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518292224:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830600.085565:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518292352:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830600.085587:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830600.085599:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830600/real 1787830600] req@ffff8b2e8803a580 x1874672518292352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830600.085610:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518292352/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830600.085618:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830600.085635:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830600.085638:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518292352:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830600.085644:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518292480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830600.085687:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830600.085699:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830600/real 1787830600] req@ffff8b2e8803a1c0 x1874672518292480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830600.085710:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518292480/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830600.085717:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830600.085720:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830600.085723:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518292480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830600.085728:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518292608:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830600.085747:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830600.085758:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830600/real 1787830600] req@ffff8b2e880392c0 x1874672518292608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830600.085769:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518292608/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830600.085777:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830600.085780:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830600.085783:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518292608:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830600.085788:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518292736:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830600.085806:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830600.085816:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830600/real 1787830600] req@ffff8b2e8803b480 x1874672518292736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830600.085825:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518292736/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830600.085838:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830600.085841:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830600.085857:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518292736:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830600.085863:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518292864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830600.085877:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830600.085884:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830600/real 1787830600] req@ffff8b2e88039e00 x1874672518292864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830600.085891:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518292864/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830625 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830600.085896:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830600.085897:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830600.085899:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518292864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830600.404765:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830600.916775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830600.916871:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830601.108808:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830601.108958:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830601.428806:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830601.940761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830601.940846:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830602.133306:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830602.133407:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830602.452791:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830602.964770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830602.964876:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830603.156780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830603.156780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830603.482340:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830603.988787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830603.989840:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830604.180793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830604.181004:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830604.500871:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830605.012755:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830605.012825:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830605.204787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830605.204862:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830605.204869:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830605.204872:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830605.204882:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3611 00000100:00080000:0.0:1787830605.204892:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830605.204911:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830605.204958:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518292992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830605.204985:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830605.204990:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830605.204991:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830605.204995:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3611 00000100:00080000:0.0:1787830605.204998:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830605.205001:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830605.205012:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830605.205022:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518293120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830605.205034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830605.205037:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830605.205039:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830605.205042:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3611 00000100:00080000:0.0:1787830605.205045:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830605.205048:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830605.205052:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830605.205062:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518293248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830605.205069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830605.205072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830605.205074:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830605.205077:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3611 00000100:00080000:0.0:1787830605.205080:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830605.205083:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830605.205087:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830605.205095:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518293376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830605.205103:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830605.205107:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830605.205108:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830605.205111:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3611 00000100:00080000:0.0:1787830605.205114:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830605.205118:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830605.205121:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830605.205129:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518293504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830605.205137:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830605.205140:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830605.205141:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830605.205144:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3611 00000100:00080000:0.0:1787830605.205146:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830605.205149:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830605.205153:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830605.205160:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518293632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830605.205168:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830605.205170:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830605.205172:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830605.205175:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3611 00000100:00080000:0.0:1787830605.205177:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830605.205180:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830605.205184:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830605.205191:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518293760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830605.205292:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518292992:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830605.205354:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830605.205397:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830605/real 1787830605] req@ffff8b2e880421c0 x1874672518292992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830605.205412:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518292992/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830605.205421:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830605.205425:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830605.205428:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518292992:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830605.205434:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518293120:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830605.205453:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830605.205464:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830605/real 1787830605] req@ffff8b2e88040f00 x1874672518293120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830605.205473:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518293120/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830605.205486:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830605.205489:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830605.205492:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518293120:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830605.205496:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518293248:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830605.205509:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830605.205517:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830605/real 1787830605] req@ffff8b2e88041a40 x1874672518293248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830605.205526:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518293248/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830605.205532:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830605.205534:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830605.205537:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518293248:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830605.205541:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518293376:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830605.205552:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830605.205559:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830605/real 1787830605] req@ffff8b2e88043480 x1874672518293376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830605.205568:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518293376/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830605.205575:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830605.205578:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830605.205581:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518293376:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830605.205585:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518293504:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830605.205602:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830605.205612:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830605/real 1787830605] req@ffff8b2e88042940 x1874672518293504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830605.205623:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518293504/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830605.205631:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830605.205634:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830605.205637:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518293504:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830605.205642:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518293632:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830605.205686:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830605.205698:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830605.205702:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830605/real 1787830605] req@ffff8b2e88040000 x1874672518293632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830605.205718:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518293632/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830605.205726:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830605.205729:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830605.205732:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518293632:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830605.205738:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518293760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830605.205757:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830605.205769:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830605/real 1787830605] req@ffff8b2e880430c0 x1874672518293760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830605.205781:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518293760/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830630 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830605.205788:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830605.205790:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830605.205793:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518293760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830605.527101:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830606.036764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830606.036937:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830606.228929:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830606.228929:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830606.548787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830607.060773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830607.060941:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830607.252762:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830607.252762:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830607.572809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830608.084776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830608.085114:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830608.277800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830608.277800:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830608.596790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830609.108909:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830609.108928:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830609.301641:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830609.302145:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830609.620794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830610.132774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830610.132877:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830610.324782:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830610.324787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830610.324796:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830610.324801:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830610.324811:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3616 00000100:00080000:3.0:1787830610.324821:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830610.324829:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830610.324883:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518293888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830610.324957:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518293888:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830610.325012:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830610.325072:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830610.325193:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830610.325198:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830610.325200:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830610.325205:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3616 00000100:00080000:3.0:1787830610.325209:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830610.325214:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830610.325224:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830610.325237:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518294016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830610.325555:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830610/real 1787830610] req@ffff8b2e8803b0c0 x1874672518293888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830610.325573:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518293888/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830610.325583:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830610.325587:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830610.325591:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518293888:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830610.325612:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518294016:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830610.325636:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830610.325669:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830610/real 1787830610] req@ffff8b2e88038f00 x1874672518294016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830610.325682:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518294016/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830610.325697:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830610.325701:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830610.325704:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518294016:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830610.325802:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830610.325806:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830610.325808:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830610.325812:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3616 00000100:00080000:3.0:1787830610.325817:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830610.325821:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830610.325827:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830610.325840:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518294144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830610.325891:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518294144:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830610.325914:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830610.325953:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830610.325957:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830610.325959:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830610.325963:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3616 00000100:00080000:3.0:1787830610.325967:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830610.325971:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830610.325976:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830610.325988:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518294272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830610.326025:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830610/real 1787830610] req@ffff8b2e880383c0 x1874672518294144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830610.326038:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518294144/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830610.326047:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830610.326051:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830610.326054:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518294144:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830610.326066:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518294272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830610.326086:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830610.326096:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830610/real 1787830610] req@ffff8b2e8803b840 x1874672518294272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830610.326108:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518294272/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830610.326116:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830610.326119:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830610.326123:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518294272:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830610.326358:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830610.326363:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830610.326365:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830610.326370:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3616 00000100:00080000:3.0:1787830610.326375:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830610.326378:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830610.326384:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830610.326466:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518294400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830610.326520:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518294400:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830610.326544:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830610.326565:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830610.326570:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830610.326572:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830610.326576:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3616 00000100:00080000:3.0:1787830610.326580:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830610.326584:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830610.326589:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830610.326601:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518294528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830610.326637:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830610/real 1787830610] req@ffff8b2e8803b840 x1874672518294400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830610.326666:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518294400/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830610.326675:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830610.326679:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830610.326682:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518294400:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830610.326694:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518294528:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830610.326714:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830610.326724:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830610/real 1787830610] req@ffff8b2e880383c0 x1874672518294528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830610.326736:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518294528/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830610.326744:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830610.326747:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830610.326750:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518294528:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830610.328379:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830610.328388:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830610.328391:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830610.328398:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3616 00000100:00080000:3.0:1787830610.328407:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830610.328412:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830610.328423:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830610.328450:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518294656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830610.329051:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518294656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830610.329101:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830610.329127:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830610/real 1787830610] req@ffff8b2e88038f00 x1874672518294656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830610.329144:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518294656/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830635 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830610.329155:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830610.329158:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830610.329162:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518294656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830610.644854:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830611.156874:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830611.157907:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830611.348885:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830611.352377:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830611.668718:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830612.181180:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830612.181527:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830612.372731:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830612.372731:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830612.692723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830613.204774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830613.205107:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830613.396856:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830613.396923:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830613.716810:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830614.228775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830614.228814:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830614.420767:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830614.420768:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830614.740772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830615.252797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830615.253023:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830615.444778:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830615.444782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830615.444791:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830615.444796:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830615.444807:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3621 00000100:00080000:3.0:1787830615.444818:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830615.444835:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830615.444942:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518294784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830615.445052:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830615.445084:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518294784:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830615.445142:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830615.445175:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830615.445179:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830615.445181:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830615.445185:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3621 00000100:00080000:3.0:1787830615.445190:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830615.445194:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830615.445203:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830615.445217:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518294912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830615.445879:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830615/real 1787830615] req@ffff8b2e8803b0c0 x1874672518294784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830615.445898:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518294784/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830615.445908:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830615.445913:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830615.445917:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518294784:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830615.445939:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518294912:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830615.445963:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830615.445975:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830615/real 1787830615] req@ffff8b2e88039e00 x1874672518294912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830615.445986:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518294912/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830615.446000:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830615.446003:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830615.446006:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518294912:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830615.446514:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830615.446520:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830615.446522:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830615.446527:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3621 00000100:00080000:3.0:1787830615.446532:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830615.446536:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830615.446544:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830615.446573:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518295040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830615.446584:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830615.446587:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830615.446589:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830615.446592:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3621 00000100:00080000:3.0:1787830615.446596:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830615.446599:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830615.446603:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830615.446613:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518295168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830615.446621:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830615.446624:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830615.446626:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830615.446629:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3621 00000100:00080000:3.0:1787830615.446632:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830615.446636:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830615.446640:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830615.446664:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518295296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830615.446674:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830615.446677:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830615.446678:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830615.446682:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3621 00000100:00080000:3.0:1787830615.446684:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830615.446688:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830615.446692:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830615.446702:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518295424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830615.446709:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830615.446712:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830615.446713:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830615.446716:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3621 00000100:00080000:3.0:1787830615.446719:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830615.446723:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830615.446727:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830615.446735:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518295552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830615.446811:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518295040:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830615.446853:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830615.446867:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830615/real 1787830615] req@ffff8b2e8803b480 x1874672518295040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830615.446879:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518295040/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830615.446887:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830615.446890:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830615.446893:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518295040:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830615.446899:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518295168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830615.446919:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830615.446929:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830615/real 1787830615] req@ffff8b2e880392c0 x1874672518295168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830615.446941:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518295168/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830615.446948:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830615.446951:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830615.446955:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518295168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830615.446961:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518295296:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830615.446978:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830615.446988:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830615/real 1787830615] req@ffff8b2e8803a1c0 x1874672518295296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830615.446999:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518295296/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830615.447007:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830615.447009:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830615.447012:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518295296:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830615.447018:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518295424:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830615.447035:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830615.447045:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830615/real 1787830615] req@ffff8b2e8803a580 x1874672518295424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830615.447055:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518295424/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830615.447062:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830615.447064:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830615.447067:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518295424:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830615.447073:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518295552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830615.447090:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830615.447100:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830615/real 1787830615] req@ffff8b2e88039a40 x1874672518295552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830615.447110:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518295552/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830640 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830615.447118:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830615.447120:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830615.447123:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518295552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830615.764782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830616.277256:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830616.277872:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830616.468840:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830616.469023:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830616.788792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830617.300748:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830617.300861:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830617.492787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830617.492787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830617.812793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830618.324785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830618.324909:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830618.516779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830618.517033:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830618.836780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830619.348775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830619.348905:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830619.540951:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830619.541077:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830619.860889:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830620.372736:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830620.373011:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830620.564807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830620.564812:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830620.564826:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830620.564830:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830620.564841:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3626 00000100:00100000:2.0:1787830620.564852:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830620.564858:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830620.564867:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830620.565256:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518295680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830620.565283:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830620.565286:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830620.565289:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830620.565293:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3626 00000100:00080000:3.0:1787830620.565297:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830620.565302:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830620.565310:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830620.565332:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518295808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830620.565335:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518295680:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830620.565344:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830620.565347:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830620.565348:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830620.565351:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3626 00000100:00080000:3.0:1787830620.565354:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830620.565356:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830620.565359:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830620.565366:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518295936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830620.565372:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830620.565375:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830620.565376:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830620.565378:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3626 00000100:00080000:3.0:1787830620.565380:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830620.565382:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830620.565384:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830620.565391:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518296064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830620.565397:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:0.0:1787830620.565398:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830620.565402:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830620.565404:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830620.565417:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3626 00000100:00080000:3.0:1787830620.565421:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830620.565425:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830620.565431:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830620.565441:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830620/real 1787830620] req@ffff8b2e88039680 x1874672518295680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830620.565445:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518296192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830620.565467:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830620.565472:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518295680/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830620.565473:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830620.565477:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830620.565485:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3626 00000100:00080000:0.0:1787830620.565491:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830620.565491:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830620.565496:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830620.565501:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787830620.565508:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830620.565514:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518295680:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830620.565517:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518296320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830620.565526:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830620.565529:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830620.565531:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830620.565535:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3626 00000100:00100000:0.0:1787830620.565539:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518295808:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830620.565539:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830620.565547:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830620.565553:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830620.565565:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830620.565567:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518296448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830620.565580:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830620/real 1787830620] req@ffff8b2e88039a40 x1874672518295808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830620.565603:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518295808/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830620.565617:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830620.565620:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830620.565623:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518295808:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830620.565628:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518295936:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830620.565645:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830620.565676:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830620/real 1787830620] req@ffff8b2e8803a580 x1874672518295936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830620.565687:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518295936/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830620.565695:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830620.565698:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830620.565700:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518295936:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830620.565706:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518296064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830620.565721:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830620.565729:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830620/real 1787830620] req@ffff8b2e8803a1c0 x1874672518296064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830620.565739:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518296064/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830620.565746:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830620.565749:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830620.565752:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518296064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830620.565757:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518296192:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830620.565772:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830620.565780:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830620/real 1787830620] req@ffff8b2e880392c0 x1874672518296192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830620.565812:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518296192/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830620.565820:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830620.565823:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830620.565826:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518296192:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830620.565831:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518296320:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830620.565847:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830620.565856:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830620/real 1787830620] req@ffff8b2e8803b480 x1874672518296320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830620.565866:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518296320/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830620.565873:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830620.565875:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830620.565878:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518296320:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830620.565894:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518296448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830620.565909:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830620.565918:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830620/real 1787830620] req@ffff8b2e88039e00 x1874672518296448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830620.565929:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518296448/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830645 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830620.565937:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830620.565939:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830620.565942:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518296448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830620.884785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830621.396794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830621.396869:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830621.588776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830621.588776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830621.908833:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830622.420876:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830622.421255:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830622.612772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830622.613072:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830622.932749:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830623.444728:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830623.444931:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830623.636793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830623.637019:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830623.956759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830624.468781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830624.468926:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830624.661129:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830624.661129:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830624.980753:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830625.492773:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830625.493133:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830625.684820:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830625.684904:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830625.685539:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830625.685550:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830625.685554:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830625.685567:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3632 00000100:00080000:1.0:1787830625.685581:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830625.685590:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830625.685638:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518296576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830625.688010:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830625.688016:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830625.688019:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830625.688029:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3632 00000100:00080000:1.0:1787830625.688040:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830625.688166:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518296576:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830625.688226:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830625.688269:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830625/real 1787830625] req@ffff8b2e88077480 x1874672518296576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830625.688293:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518296576/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830625.688306:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830625.688311:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830625.688316:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518296576:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830625.688383:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830625.688397:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830625.688430:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518296704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830625.688465:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830625.688470:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830625.688472:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830625.688484:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3632 00000100:00080000:1.0:1787830625.688490:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830625.688496:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830625.688502:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830625.688515:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518296832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830625.688527:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830625.688530:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830625.688532:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830625.688535:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3632 00000100:00080000:1.0:1787830625.688539:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830625.688543:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830625.688834:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830625.688849:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518296960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830625.688859:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830625.688861:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830625.688863:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830625.688867:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3632 00000100:00080000:1.0:1787830625.688871:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830625.688874:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830625.688879:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830625.688888:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518297088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830625.688896:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830625.688899:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830625.688901:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830625.688904:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3632 00000100:00080000:1.0:1787830625.688907:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830625.688910:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830625.688914:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830625.688922:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518297216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830625.688929:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830625.688932:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830625.688933:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830625.688936:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3632 00000100:00080000:1.0:1787830625.688939:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830625.688942:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830625.688945:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830625.688953:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518297344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830625.689027:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518296704:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830625.689167:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830625.689183:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830625/real 1787830625] req@ffff8b2e88075680 x1874672518296704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830625.689196:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518296704/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830625.689210:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830625.689214:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830625.689217:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518296704:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830625.689223:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518296832:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830625.689244:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830625.689255:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830625/real 1787830625] req@ffff8b2e88074780 x1874672518296832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830625.689266:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518296832/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830625.689273:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830625.689276:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830625.689279:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518296832:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830625.689285:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518296960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830625.689302:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830625.689312:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830625/real 1787830625] req@ffff8b2e88074f00 x1874672518296960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830625.689324:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518296960/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830625.689331:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830625.689334:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830625.689337:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518296960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830625.689342:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518297088:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830625.689361:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830625.689371:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830625/real 1787830625] req@ffff8b2e88075a40 x1874672518297088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830625.689381:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518297088/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830625.689389:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830625.689392:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830625.689395:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518297088:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830625.689400:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518297216:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830625.689418:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830625.689428:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830625/real 1787830625] req@ffff8b2e88076d00 x1874672518297216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830625.689439:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518297216/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830625.689447:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830625.689450:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830625.689452:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518297216:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830625.689457:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518297344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830625.689474:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830625.689490:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830625/real 1787830625] req@ffff8b2e88077c00 x1874672518297344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830625.689501:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518297344/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830650 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830625.689509:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830625.689512:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830625.689515:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518297344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830626.005136:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830626.516787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830626.516907:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830626.708865:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830626.712023:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830627.031436:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830627.540784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830627.541018:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830627.734387:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830627.735538:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830628.057435:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830628.564900:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830628.565371:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830628.758577:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830628.758664:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830629.076786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830629.588805:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830629.589143:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830629.780749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830629.780800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830630.100882:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830630.612928:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830630.613073:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830630.804813:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830630.804882:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830630.804889:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830630.804894:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830630.804904:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3637 00000100:00100000:2.0:1787830630.804907:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830630.804915:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830630.804925:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830630.804970:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518297472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830630.805048:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518297472:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830630.805109:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830630.805142:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830630.805146:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830630.805148:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830630.805153:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3637 00000100:00080000:3.0:1787830630.805157:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830630.805161:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830630.805171:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830630.805186:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518297600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830630.805403:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830630/real 1787830630] req@ffff8b2e8803b0c0 x1874672518297472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830630.805431:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518297472/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830630.805441:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830630.805445:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830630.805448:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518297472:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830630.805470:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518297600:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830630.805715:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830630.805912:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830630.805916:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830630.805918:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830630.805923:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3637 00000100:00080000:3.0:1787830630.805927:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830630.805931:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830630.805937:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830630.805952:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518297728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830630.805962:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830630.805966:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830630.805967:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830630.805970:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3637 00000100:00080000:3.0:1787830630.805973:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830630.805977:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830630.805981:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830630.805991:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518297856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830630.806000:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830630.806003:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830630.806005:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830630.806009:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3637 00000100:00080000:3.0:1787830630.806012:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830630.806099:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830630/real 1787830630] req@ffff8b2e88038f00 x1874672518297600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830630.806119:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518297600/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830630.806129:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830630.806132:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830630.806136:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518297600:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830630.806149:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518297728:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830630.806173:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830630.806185:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830630/real 1787830630] req@ffff8b2e880383c0 x1874672518297728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830630.806197:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518297728/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830630.806206:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830630.806209:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830630.806212:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518297728:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830630.806218:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518297856:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830630.806237:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830630.806247:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830630/real 1787830630] req@ffff8b2e8803b840 x1874672518297856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830630.806258:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518297856/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830630.806270:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830630.806273:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830630.806277:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518297856:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830630.806319:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830630.806324:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830630.806338:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518297984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830630.806356:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830630.806360:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830630.806362:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830630.806365:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3637 00000100:00080000:3.0:1787830630.806369:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830630.806373:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830630.806378:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830630.806388:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518298112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830630.806396:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830630.806399:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830630.806401:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830630.806404:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3637 00000100:00080000:3.0:1787830630.806527:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830630.806535:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830630.806539:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830630.806552:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518298240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830630.806616:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518297984:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830630.806715:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830630.806825:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830630/real 1787830630] req@ffff8b2e8803b840 x1874672518297984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830630.806838:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518297984/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830630.806847:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830630.806850:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830630.806854:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518297984:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830630.806860:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518298112:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830630.806882:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830630.806893:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830630/real 1787830630] req@ffff8b2e880383c0 x1874672518298112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830630.806905:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518298112/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830630.806912:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830630.806915:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830630.806919:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518298112:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830630.806925:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518298240:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830630.806943:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830630.806954:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830630/real 1787830630] req@ffff8b2e88038f00 x1874672518298240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830630.806965:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518298240/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830655 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830630.806973:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830630.806976:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830630.806978:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518298240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830631.124840:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830631.636779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830631.636887:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830631.828815:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830631.828972:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830632.148841:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830632.661303:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830632.661794:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830632.852734:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830632.853172:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830633.173004:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830633.685589:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830633.685860:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830633.876806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830633.876900:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830634.196830:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830634.710178:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830634.710252:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830634.900885:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830634.902859:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830635.221367:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830635.732855:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830635.732988:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:3.0:1787830635.924786:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830635.924797:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830635.924801:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830635.924803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830635.924812:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3642 00000100:00080000:3.0:1787830635.924823:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830635.924830:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830635.924868:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518298368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830635.924893:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830635.924950:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830635.924952:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830635.924956:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3642 00000100:00080000:3.0:1787830635.924960:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830635.924964:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830635.924973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830635.924987:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518298496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830635.924997:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830635.925000:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830635.925002:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830635.925006:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3642 00000100:00080000:3.0:1787830635.925009:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830635.925012:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830635.925017:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830635.925026:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518298624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830635.925034:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830635.925037:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830635.925038:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830635.925041:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3642 00000100:00080000:3.0:1787830635.925044:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830635.925047:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830635.925052:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830635.925061:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518298752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830635.925069:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830635.925072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830635.925073:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830635.925076:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3642 00000100:00080000:3.0:1787830635.925080:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830635.925083:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830635.925087:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830635.925097:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518298880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830635.925106:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830635.925108:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830635.925110:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830635.925114:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3642 00000100:00080000:3.0:1787830635.925117:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830635.925119:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830635.925123:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830635.925133:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518299008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830635.925141:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830635.925144:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830635.925145:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830635.925149:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3642 00000100:00080000:3.0:1787830635.925152:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830635.925155:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830635.925159:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830635.925168:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518299136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830635.925279:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518298368:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830635.925336:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830635.925376:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830635/real 1787830635] req@ffff8b2e8803b0c0 x1874672518298368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830635.925379:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:0.0:1787830635.925396:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518298368/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830635.925406:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830635.925410:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830635.925413:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518298368:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830635.925420:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518298496:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830635.925439:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830635.925449:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830635/real 1787830635] req@ffff8b2e88039e00 x1874672518298496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830635.925459:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518298496/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830635.925466:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830635.925469:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830635.925472:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518298496:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830635.925476:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518298624:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830635.925493:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830635.925502:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830635/real 1787830635] req@ffff8b2e8803b480 x1874672518298624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830635.925513:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518298624/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830635.925521:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830635.925524:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830635.925527:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518298624:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830635.925532:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518298752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830635.925551:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830635.925561:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830635/real 1787830635] req@ffff8b2e880392c0 x1874672518298752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830635.925572:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518298752/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830635.925579:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830635.925582:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830635.925585:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518298752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830635.925591:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518298880:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830635.925607:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830635.925617:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830635/real 1787830635] req@ffff8b2e8803a1c0 x1874672518298880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830635.925628:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518298880/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830635.925635:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830635.925638:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830635.925641:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518298880:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830635.925664:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518299008:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830635.925693:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830635.925705:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830635/real 1787830635] req@ffff8b2e8803a580 x1874672518299008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830635.925717:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518299008/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830635.925729:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830635.925732:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830635.925735:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518299008:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830635.925740:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518299136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830635.925757:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830635.925766:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830635/real 1787830635] req@ffff8b2e88039a40 x1874672518299136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830635.925775:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518299136/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830660 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830635.925782:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830635.925785:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830635.925788:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518299136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830636.244802:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830636.756877:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830636.756889:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830636.949039:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830636.949312:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830637.268790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830637.780797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830637.781822:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830637.972844:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830637.972951:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830638.292783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830638.804805:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830638.806544:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830638.996964:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830638.997109:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830639.316796:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830639.829071:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830639.829084:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830640.020788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830640.022493:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830640.340796:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830640.852744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830640.852888:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830641.044790:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830641.044846:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830641.044851:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830641.044852:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830641.044857:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830641.044868:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3647 00000100:00080000:1.0:1787830641.044880:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830641.044887:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830641.044922:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518299264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830641.044959:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830641.044962:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830641.044964:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830641.044967:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3647 00000100:00080000:1.0:1787830641.044971:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830641.044975:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830641.044984:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830641.044997:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518299392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830641.045006:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830641.045009:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830641.045011:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830641.045014:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3647 00000100:00080000:1.0:1787830641.045017:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830641.045020:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830641.045024:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830641.045032:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518299520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830641.045041:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830641.045164:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830641.045167:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830641.045171:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3647 00000100:00080000:1.0:1787830641.045175:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830641.045179:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830641.045184:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830641.045195:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518299648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830641.045204:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830641.045207:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830641.045208:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830641.045212:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3647 00000100:00080000:1.0:1787830641.045214:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830641.045218:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830641.045221:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830641.045229:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518299776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830641.045236:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830641.045239:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830641.045240:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830641.045243:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3647 00000100:00080000:1.0:1787830641.045246:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830641.045249:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830641.045253:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830641.045262:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518299904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830641.045275:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830641.045278:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830641.045279:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830641.045282:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3647 00000100:00080000:1.0:1787830641.045285:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830641.045289:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830641.045293:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830641.045302:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518300032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830641.045378:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518299264:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830641.045430:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830641.045471:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830641/real 1787830641] req@ffff8b2e88076940 x1874672518299264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830641.045486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518299264/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830641.045496:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830641.045500:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830641.045504:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518299264:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830641.045510:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518299392:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830641.045533:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830641.045546:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830641/real 1787830641] req@ffff8b2e88077c00 x1874672518299392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830641.045633:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518299392/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830641.045642:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830641.045679:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830641.045684:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518299392:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830641.045690:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518299520:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830641.045712:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830641.045723:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830641/real 1787830641] req@ffff8b2e88076d00 x1874672518299520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830641.045734:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518299520/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830641.045742:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830641.045745:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830641.045748:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518299520:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830641.045753:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518299648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830641.045770:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830641.045780:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830641/real 1787830641] req@ffff8b2e88075a40 x1874672518299648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830641.045790:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518299648/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830641.045796:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830641.045799:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830641.045803:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518299648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830641.045808:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518299776:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830641.045826:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830641.045835:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830641/real 1787830641] req@ffff8b2e88074f00 x1874672518299776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830641.045845:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518299776/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830641.045852:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830641.045855:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830641.045857:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518299776:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830641.045863:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518299904:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830641.045880:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830641.045889:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830641/real 1787830641] req@ffff8b2e88074780 x1874672518299904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830641.045899:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518299904/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830641.045910:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830641.045912:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830641.045916:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518299904:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830641.045921:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518300032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830641.045938:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830641.045947:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830641/real 1787830641] req@ffff8b2e88075680 x1874672518300032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830641.045957:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518300032/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830666 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830641.045964:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830641.045966:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830641.045969:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518300032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830641.364768:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830641.881500:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830641.881797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830642.068784:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830642.068784:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830642.389209:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830642.901025:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830642.901207:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830643.092772:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830643.092809:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830643.414557:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830643.924775:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830643.927753:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830644.116774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830644.116774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830644.436799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830644.948966:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830644.949798:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830645.140803:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830645.141061:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830645.460815:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830645.972782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830645.973630:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830646.164790:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830646.164803:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830646.164807:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830646.164817:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3652 00000100:00080000:1.0:1787830646.164829:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830646.164836:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830646.164875:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518300160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830646.164902:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830646.164906:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830646.164908:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830646.164912:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3652 00000100:00080000:1.0:1787830646.164915:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830646.164919:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830646.164929:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830646.164940:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518300288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830646.164948:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830646.164952:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830646.164953:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830646.164957:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3652 00000100:00080000:1.0:1787830646.164959:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830646.164962:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830646.164966:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830646.164974:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518300416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830646.164982:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830646.164985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830646.164987:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830646.164990:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3652 00000100:00080000:1.0:1787830646.164993:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830646.164996:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830646.165000:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830646.165012:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518300544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830646.165021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830646.165024:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830646.165026:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830646.165029:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3652 00000100:00080000:1.0:1787830646.165032:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830646.165036:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830646.165039:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830646.165048:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518300672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830646.165057:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830646.165060:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830646.165061:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830646.165065:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3652 00000100:00080000:1.0:1787830646.165068:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830646.165071:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830646.165075:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830646.165084:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518300800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830646.165092:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830646.165095:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830646.165097:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830646.165100:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3652 00000100:00080000:1.0:1787830646.165103:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830646.165106:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830646.165110:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830646.165117:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518300928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830646.165213:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830646.165630:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518300160:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830646.165688:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000800:00000100:3.0:1787830646.165711:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830646.165757:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830646/real 1787830646] req@ffff8b2e88077480 x1874672518300160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830646.165774:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518300160/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830646.165789:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830646.165793:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830646.165797:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518300160:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830646.165803:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518300288:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830646.165825:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830646.165838:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830646/real 1787830646] req@ffff8b2e88077840 x1874672518300288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830646.165850:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518300288/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830646.165859:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830646.165862:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830646.165871:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518300288:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830646.165877:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518300416:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830646.165906:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830646.165917:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830646/real 1787830646] req@ffff8b2e880770c0 x1874672518300416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830646.165928:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518300416/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830646.165936:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830646.165939:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830646.165942:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518300416:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830646.165949:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518300544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830646.165966:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830646.165975:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830646/real 1787830646] req@ffff8b2e880761c0 x1874672518300544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830646.165985:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518300544/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830646.165991:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830646.165994:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830646.165996:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518300544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830646.166001:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518300672:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830646.166015:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830646.166024:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830646/real 1787830646] req@ffff8b2e88074000 x1874672518300672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830646.166033:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518300672/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830646.166039:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830646.166042:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830646.166045:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518300672:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830646.166050:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518300800:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830646.166064:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830646.166073:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830646/real 1787830646] req@ffff8b2e88074b40 x1874672518300800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830646.166082:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518300800/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830646.166090:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830646.166092:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830646.166095:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518300800:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830646.166100:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518300928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830646.166116:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830646.166124:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830646/real 1787830646] req@ffff8b2e88075e00 x1874672518300928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830646.166133:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518300928/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830671 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830646.166140:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830646.166142:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830646.166145:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518300928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830646.484764:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830646.996761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830646.996843:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830647.188780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830647.188780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830647.508772:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830648.020771:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830648.020814:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830648.212807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830648.212869:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830648.532935:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830649.045327:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830649.045629:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830649.236848:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830649.236852:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830649.556883:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830650.068774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830650.072373:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830650.260788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830650.260788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830650.580794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830651.092865:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830651.093064:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830651.284796:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830651.284808:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830651.284812:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830651.284822:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3657 00000100:00080000:1.0:1787830651.284833:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830651.284839:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830651.284879:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518301056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830651.284906:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830651.284909:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830651.284911:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830651.284914:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3657 00000100:00080000:1.0:1787830651.284918:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830651.284921:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830651.284931:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830651.284946:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518301184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830651.284955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830651.284958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830651.284960:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830651.284963:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3657 00000100:00080000:1.0:1787830651.284966:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830651.284969:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830651.284973:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830651.284982:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518301312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830651.284990:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830651.284993:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830651.284995:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830651.284998:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3657 00000100:00080000:1.0:1787830651.285001:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830651.285004:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830651.285007:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830651.285017:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518301440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830651.285025:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830651.285028:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830651.285030:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830651.285033:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3657 00000100:00080000:1.0:1787830651.285036:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830651.285040:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830651.285044:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830651.285053:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518301568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830651.285061:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830651.285064:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830651.285065:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830651.285069:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3657 00000100:00080000:1.0:1787830651.285072:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830651.285075:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830651.285079:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830651.285089:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518301696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830651.285097:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830651.285100:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830651.285101:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830651.285104:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3657 00000100:00080000:1.0:1787830651.285106:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830651.285109:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830651.285113:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830651.285120:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518301824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830651.285249:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518301056:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830651.285305:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830651.285351:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830651/real 1787830651] req@ffff8b2e88076580 x1874672518301056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830651.285365:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518301056/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830651.285375:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830651.285379:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830651.285382:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518301056:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830651.285388:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518301184:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830651.285409:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830651.285421:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830651/real 1787830651] req@ffff8b2e88075e00 x1874672518301184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830651.285431:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518301184/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830651.285439:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830651.285443:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830651.285445:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518301184:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830651.285451:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518301312:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830651.285473:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830651.285486:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830651/real 1787830651] req@ffff8b2e88074b40 x1874672518301312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830651.285504:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518301312/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830651.285512:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830651.285515:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830651.285520:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518301312:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830651.285522:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830651.285528:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518301440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830651.285548:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830651.285561:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830651/real 1787830651] req@ffff8b2e88074000 x1874672518301440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830651.285573:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518301440/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830651.285595:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830651.285598:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830651.285601:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518301440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830651.285606:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518301568:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830651.285623:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000800:00000100:3.0:1787830651.285624:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830651.285636:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830651/real 1787830651] req@ffff8b2e880761c0 x1874672518301568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830651.285671:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518301568/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830651.285679:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830651.285682:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830651.285685:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518301568:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830651.285691:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518301696:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830651.285709:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830651.285719:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830651/real 1787830651] req@ffff8b2e880770c0 x1874672518301696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830651.285730:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518301696/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830651.285737:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830651.285739:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830651.285742:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518301696:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830651.285748:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518301824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830651.285765:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830651.285773:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830651/real 1787830651] req@ffff8b2e88077840 x1874672518301824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830651.285780:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518301824/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830676 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830651.285788:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830651.285790:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830651.285792:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518301824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830651.605193:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830652.116800:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830652.116972:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830652.308794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830652.308874:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830652.629459:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830653.140766:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830653.141761:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830653.332733:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830653.335132:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830653.653327:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830654.164982:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830654.165323:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830654.356786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830654.356881:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830654.676783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830655.188774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830655.188810:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830655.381349:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830655.381794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830655.700789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830656.212791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830656.213086:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830656.404740:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830656.404806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787830656.405378:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830656.405384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830656.405388:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830656.405396:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3662 00000100:00080000:2.0:1787830656.405406:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830656.405413:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830656.405447:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518301952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830656.405476:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830656.405480:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830656.405482:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830656.405485:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3662 00000100:00080000:2.0:1787830656.405488:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830656.405491:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830656.405500:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830656.405509:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518302080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830656.405518:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830656.405521:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830656.405523:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830656.405525:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3662 00000100:00080000:2.0:1787830656.405528:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830656.405531:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830656.405535:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830656.405552:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518302208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830656.405561:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830656.405564:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830656.405566:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830656.405569:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3662 00000100:00080000:2.0:1787830656.405571:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830656.405574:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830656.405578:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830656.405587:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518302336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830656.405596:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830656.405598:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830656.405600:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830656.405603:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3662 00000100:00080000:2.0:1787830656.405605:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830656.405608:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830656.405611:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830656.405618:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518302464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830656.405625:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830656.405628:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830656.405629:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830656.405632:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3662 00000100:00080000:2.0:1787830656.405635:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830656.405637:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830656.405641:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830656.405669:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518302592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830656.405677:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830656.405679:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830656.405681:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830656.405684:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3662 00000100:00080000:2.0:1787830656.405686:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830656.405690:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830656.405694:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830656.405703:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518302720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830656.405712:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518301952:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830656.405816:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518302080:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830656.405846:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518302208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830656.405870:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518302336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830656.405891:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518302464:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830656.405910:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518302592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830656.405934:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518302720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830656.725245:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830657.236774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830657.236820:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830657.428774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830657.428875:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830657.748787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830658.260774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830658.262034:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830658.455732:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830658.456774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830658.774687:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830659.284790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830659.285102:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830659.476794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830659.476794:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000400:00000100:1.0:1787830659.476959:0:2335:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:1.0:1787830659.477004:0:2335:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:1.0:1787830659.477019:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830659.477050:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830659.477068:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830659.477077:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830659.477086:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787830659.477091:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830656/real 1787830659] req@ffff8b2e8801e580 x1874672518301952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000800:00000100:1.0:1787830659.477096:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:1.0:1787830659.477107:0:2335:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:3.0:1787830659.477119:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518301952/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830659.477129:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830659.477131:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830659.477134:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518301952:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830659.477138:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830656/real 1787830659] req@ffff8b2e8801e940 x1874672518302080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830659.477146:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518302080/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830659.477151:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830659.477153:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830659.477168:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518302080:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830659.477175:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830656/real 1787830659] req@ffff8b2e8801f480 x1874672518302208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830659.477186:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518302208/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830659.477193:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830659.477196:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830659.477199:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518302208:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830659.477204:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830656/real 1787830659] req@ffff8b2e8801cf00 x1874672518302336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830659.477215:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518302336/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830659.477222:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830659.477224:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830659.477228:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518302336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830659.477233:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830656/real 1787830659] req@ffff8b2e8801c000 x1874672518302464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830659.477244:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518302464/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830659.477251:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830659.477255:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830659.477258:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518302464:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830659.477264:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830656/real 1787830659] req@ffff8b2e8801da40 x1874672518302592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830659.477274:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518302592/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830659.477281:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830659.477285:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830659.477288:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518302592:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830659.477314:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830656/real 1787830659] req@ffff8b2e8801fc00 x1874672518302720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830659.477326:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518302720/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830681 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830659.477333:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830659.477336:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830659.477338:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518302720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830659.796792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830660.308838:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830660.309323:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830660.500801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830660.504051:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830660.820759:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830661.333030:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830661.334219:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830661.524749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830661.524801:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787830661.524978:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830661.524983:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830661.524988:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830661.524995:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3667 00000100:00080000:2.0:1787830661.525004:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830661.525012:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830661.525046:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801fc00 x1874672518302848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830661.525077:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830661.525080:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830661.525082:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830661.525086:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3667 00000100:00080000:2.0:1787830661.525089:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830661.525093:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830661.525106:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830661.525119:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801da40 x1874672518302976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830661.525127:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830661.525130:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830661.525132:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830661.525135:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3667 00000100:00080000:2.0:1787830661.525138:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830661.525142:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830661.525147:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830661.525156:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c000 x1874672518303104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830661.525169:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830661.525172:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830661.525174:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830661.525177:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3667 00000100:00080000:2.0:1787830661.525180:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830661.525183:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830661.525187:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830661.525196:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cf00 x1874672518303232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830661.525202:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830661.525205:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830661.525206:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830661.525209:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3667 00000100:00080000:2.0:1787830661.525212:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830661.525215:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830661.525219:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830661.525227:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f480 x1874672518303360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830661.525234:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830661.525236:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830661.525238:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830661.525241:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3667 00000100:00080000:2.0:1787830661.525244:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830661.525248:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830661.525252:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830661.525260:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e940 x1874672518303488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830661.525267:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830661.525270:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830661.525272:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830661.525275:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3667 00000100:00080000:2.0:1787830661.525278:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830661.525280:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830661.525284:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830661.525291:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801e580 x1874672518303616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830661.525542:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518302848:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830661.525599:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830661.525637:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830661/real 1787830661] req@ffff8b2e8801fc00 x1874672518302848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830661.525674:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801fc00 x1874672518302848/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830661.525684:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830661.525688:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830661.525691:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801fc00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518302848:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830661.525699:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518302976:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830661.525721:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830661.525733:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830661/real 1787830661] req@ffff8b2e8801da40 x1874672518302976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830661.525744:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801da40 x1874672518302976/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830661.525752:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830661.525755:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830661.525758:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801da40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518302976:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830661.525763:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518303104:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830661.528211:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830661.528245:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830661/real 1787830661] req@ffff8b2e8801c000 x1874672518303104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830661.528262:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c000 x1874672518303104/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830661.528272:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830661.528276:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830661.528279:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518303104:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830661.528289:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518303232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830661.528311:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830661.528323:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830661/real 1787830661] req@ffff8b2e8801cf00 x1874672518303232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830661.528335:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cf00 x1874672518303232/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830661.528343:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830661.528346:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830661.528348:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cf00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518303232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830661.528354:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518303360:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830661.528372:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830661.528382:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830661/real 1787830661] req@ffff8b2e8801f480 x1874672518303360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830661.528393:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f480 x1874672518303360/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830661.528401:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830661.528404:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830661.528407:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518303360:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830661.528413:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518303488:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830661.528431:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830661.528443:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830661/real 1787830661] req@ffff8b2e8801e940 x1874672518303488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830661.528452:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e940 x1874672518303488/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830661.528464:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830661.528467:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830661.528470:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518303488:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830661.528477:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518303616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830661.528494:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830661.528515:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830661/real 1787830661] req@ffff8b2e8801e580 x1874672518303616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830661.528527:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801e580 x1874672518303616/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830686 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830661.528533:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830661.528537:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830661.528540:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801e580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518303616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830661.854933:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830662.356871:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830662.356913:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830662.548797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830662.548906:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830662.868787:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830663.380794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830663.380833:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830663.573038:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830663.574143:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830663.893341:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830664.404744:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830664.404865:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830664.596755:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830664.596852:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830664.916846:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830665.429007:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830665.429136:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830665.620745:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830665.623615:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830665.940779:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830666.452895:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830666.454023:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830666.644788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830666.644788:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830666.645095:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830666.645101:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830666.645104:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830666.645113:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3673 00000100:00080000:1.0:1787830666.645124:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830666.645131:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830666.645171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518303744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830666.645200:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830666.645203:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830666.645205:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830666.645209:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3673 00000100:00080000:1.0:1787830666.645213:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830666.645218:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830666.645228:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830666.645240:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518303744:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830666.645254:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518303872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830666.645265:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830666.645278:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830666.645280:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830666.645284:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3673 00000100:00080000:1.0:1787830666.645289:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830666.645292:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000800:00000100:3.0:1787830666.645297:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000008:00100000:1.0:1787830666.645301:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830666.645316:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518304000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830666.645325:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830666.645328:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830666.645329:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830666.645333:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3673 00000100:00080000:1.0:1787830666.645337:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830666.645341:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830666/real 1787830666] req@ffff8b2e88077480 x1874672518303744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830666.645345:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830666.645352:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830666.645364:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518303744/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830666.645368:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518304128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830666.645388:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830666.645390:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830666.645394:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787830666.645396:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830666.645401:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830666.645406:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3673 00000100:00080000:1.0:1787830666.645410:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830666.645410:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518303744:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830666.645418:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830666.645423:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830666.645435:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518304256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830666.645443:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830666.645445:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830666.645447:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830666.645450:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3673 00000100:00080000:1.0:1787830666.645454:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830666.645456:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830666.645460:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830666.645479:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518304384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830666.645488:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830666.645491:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830666.645492:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830666.645495:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3673 00000100:00080000:1.0:1787830666.645499:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830666.645502:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830666.645506:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830666.645516:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518304512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830666.645895:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518303872:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830666.645924:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830666.645938:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830666/real 1787830666] req@ffff8b2e88075680 x1874672518303872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830666.645949:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518303872/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830666.645957:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830666.645961:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830666.645964:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518303872:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830666.645970:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518304000:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830666.645991:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830666.646004:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830666/real 1787830666] req@ffff8b2e88074780 x1874672518304000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830666.646015:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518304000/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830666.646023:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830666.646026:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830666.646029:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518304000:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830666.646046:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518304128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830666.646065:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830666.646075:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830666/real 1787830666] req@ffff8b2e88074f00 x1874672518304128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830666.646085:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518304128/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830666.646093:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830666.646096:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830666.646099:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518304128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830666.646104:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518304256:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830666.646123:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830666.646133:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830666/real 1787830666] req@ffff8b2e88075a40 x1874672518304256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830666.646150:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518304256/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830666.646159:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830666.646161:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830666.646165:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518304256:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830666.646170:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518304384:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830666.646185:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830666.646194:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830666/real 1787830666] req@ffff8b2e88076d00 x1874672518304384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830666.646204:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518304384/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830666.646211:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830666.646214:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830666.646217:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518304384:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830666.646221:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518304512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830666.646298:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830666.646309:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830666/real 1787830666] req@ffff8b2e88077c00 x1874672518304512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830666.646320:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518304512/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830691 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830666.646334:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830666.646337:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830666.646340:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518304512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830666.964800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830667.476770:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830667.477114:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830667.668787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830667.669160:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830667.988786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830668.500930:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830668.501670:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830668.692806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830668.692912:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830669.012790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830669.524767:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830669.525955:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830669.716796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830669.716809:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830670.036824:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830670.549132:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830670.549268:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830670.740794:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830670.740828:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830671.060793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830671.572936:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830671.573944:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830671.764767:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830671.764769:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830671.764776:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830671.764780:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830671.764790:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3678 00000100:00100000:0.0:1787830671.764793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830671.764800:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830671.764808:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830671.764846:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518304640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830671.764873:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830671.764877:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830671.764879:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830671.764883:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3678 00000100:00080000:1.0:1787830671.764887:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830671.764890:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830671.764899:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830671.764912:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518304768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830671.764920:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830671.764923:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830671.764925:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830671.764928:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3678 00000100:00080000:1.0:1787830671.764931:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830671.764934:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830671.764938:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830671.764947:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518304896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830671.764955:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830671.764958:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830671.764959:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830671.764963:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3678 00000100:00080000:1.0:1787830671.764965:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830671.764972:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518304640:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830671.764979:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830671.764983:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830671.764992:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518305024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830671.765000:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787830671.765032:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830671.765042:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830671.765043:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830671.765047:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3678 00000100:00080000:1.0:1787830671.765050:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830671.765053:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830671.765057:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830671.765067:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518305152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830671.765073:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830671/real 1787830671] req@ffff8b2e88076940 x1874672518304640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830671.765075:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830671.765079:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830671.765080:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830671.765083:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3678 00000100:00080000:1.0:1787830671.765086:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830671.765088:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518304640/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830671.765089:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830671.765093:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787830671.765098:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:1.0:1787830671.765101:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518305280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830671.765101:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830671.765105:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518304640:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830671.765109:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830671.765112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:3.0:1787830671.765112:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518304768:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830671.765113:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830671.765116:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3678 00000100:00080000:1.0:1787830671.765119:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830671.765122:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830671.765126:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:3.0:1787830671.765132:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830671.765134:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518305408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830671.765144:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830671/real 1787830671] req@ffff8b2e88077c00 x1874672518304768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830671.765154:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518304768/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830671.765163:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830671.765165:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830671.765169:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518304768:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830671.765175:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518304896:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830671.765194:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830671.765205:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830671/real 1787830671] req@ffff8b2e88076d00 x1874672518304896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830671.765216:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518304896/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830671.765224:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830671.765226:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830671.765229:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518304896:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830671.765253:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518305024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830671.765271:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830671.765283:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830671/real 1787830671] req@ffff8b2e88075a40 x1874672518305024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830671.765294:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518305024/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830671.765302:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830671.765304:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830671.765307:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518305024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830671.765312:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518305152:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830671.765331:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830671.765342:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830671/real 1787830671] req@ffff8b2e88074f00 x1874672518305152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830671.765353:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518305152/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830671.765361:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830671.765364:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830671.765367:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518305152:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830671.765372:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518305280:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830671.765390:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830671.765399:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830671/real 1787830671] req@ffff8b2e88074780 x1874672518305280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830671.765422:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518305280/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830671.765429:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830671.765432:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830671.765444:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518305280:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830671.765449:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518305408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830671.765466:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830671.765476:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830671/real 1787830671] req@ffff8b2e88075680 x1874672518305408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830671.765486:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518305408/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830696 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830671.765493:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830671.765496:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830671.765498:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518305408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830672.084842:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830672.596801:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830672.596939:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830672.788779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830672.788779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830673.108804:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830673.620776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830673.620828:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830673.812934:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830673.813222:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830674.133551:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830674.644824:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830674.645675:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830674.836866:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830674.837047:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830675.156794:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830675.668780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830675.669012:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830675.860783:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830675.860891:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830676.180839:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830676.692950:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830676.698719:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830676.884748:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830676.884750:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830676.884751:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830676.884761:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830676.884766:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830676.884776:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3683 00000100:00080000:1.0:1787830676.884787:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830676.884795:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830676.884831:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518305536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830676.884860:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830676.884863:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830676.884865:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830676.884869:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3683 00000100:00080000:1.0:1787830676.884872:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830676.884876:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830676.884885:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830676.884899:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518305664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830676.884908:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830676.884913:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830676.884915:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830676.884918:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3683 00000100:00080000:1.0:1787830676.884922:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830676.884925:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830676.884929:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830676.884939:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518305792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830676.884948:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830676.884951:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830676.884952:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830676.884956:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3683 00000100:00080000:1.0:1787830676.884959:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830676.884962:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830676.884966:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830676.884975:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518305920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830676.884983:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830676.884986:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830676.884988:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830676.884991:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3683 00000100:00080000:1.0:1787830676.884994:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830676.884997:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830676.885002:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830676.885011:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518306048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830676.885019:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830676.885022:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830676.885023:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830676.885027:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3683 00000100:00080000:1.0:1787830676.885030:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830676.885033:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830676.885037:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830676.885046:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518306176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830676.885054:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830676.885057:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830676.885058:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830676.885061:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3683 00000100:00080000:1.0:1787830676.885064:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830676.885067:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830676.885071:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830676.885079:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518306304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830676.885151:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518305536:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830676.885209:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830676.885252:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830676/real 1787830676] req@ffff8b2e88077480 x1874672518305536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830676.885267:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518305536/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830676.885276:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830676.885280:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830676.885283:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518305536:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830676.885289:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518305664:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830676.885311:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830676.885325:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830676/real 1787830676] req@ffff8b2e88077840 x1874672518305664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830676.885337:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518305664/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830676.885345:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830676.885347:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830676.885350:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518305664:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830676.885355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518305792:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830676.885376:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830676.885387:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830676/real 1787830676] req@ffff8b2e880770c0 x1874672518305792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830676.885410:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518305792/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830676.885418:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830676.885421:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830676.885424:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518305792:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830676.885430:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518305920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830676.885449:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830676.885460:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830676/real 1787830676] req@ffff8b2e880761c0 x1874672518305920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830676.885472:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518305920/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830676.885479:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830676.885482:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830676.885484:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518305920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830676.885490:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518306048:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830676.885507:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830676.885517:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830676/real 1787830676] req@ffff8b2e88074000 x1874672518306048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830676.885534:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518306048/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830676.885542:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830676.885544:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830676.885547:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518306048:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830676.885552:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518306176:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830676.885570:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830676.885579:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830676/real 1787830676] req@ffff8b2e88074b40 x1874672518306176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830676.885590:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518306176/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830676.885597:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830676.885600:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830676.885603:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518306176:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830676.885609:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518306304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830676.885627:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830676.885637:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830676/real 1787830676] req@ffff8b2e88075e00 x1874672518306304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830676.885673:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518306304/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830701 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830676.885686:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830676.885689:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830676.885692:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518306304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830677.207157:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830677.716779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830677.717288:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830677.908896:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830677.909005:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830678.229184:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830678.740778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830678.740919:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830678.932791:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830678.932997:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830679.252793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830679.764874:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830679.764972:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830679.956800:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830679.956974:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830680.276932:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830680.788927:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830680.788971:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830680.980813:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830680.980815:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830681.300774:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830681.812823:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830681.813269:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830682.004774:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830682.004779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830682.004788:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830682.004794:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830682.004815:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3688 00000100:00080000:1.0:1787830682.004834:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830682.004842:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830682.004883:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518306432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830682.004913:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830682.004916:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830682.004918:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830682.004922:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3688 00000100:00080000:1.0:1787830682.004926:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830682.004930:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830682.004948:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830682.004952:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518306432:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830682.004966:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518306560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830682.004977:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830682.004981:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830682.004983:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830682.004987:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3688 00000100:00080000:1.0:1787830682.004992:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830682.004996:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830682.005001:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830682.005015:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518306688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830682.005015:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830682.005032:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830682.005035:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830682.005037:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830682.005041:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3688 00000100:00080000:1.0:1787830682.005045:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830682.005049:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830682.005054:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830682.005065:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830682/real 1787830682] req@ffff8b2e88076580 x1874672518306432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830682.005066:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518306816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830682.005094:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830682.005097:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830682.005099:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:3.0:1787830682.005104:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518306432/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830682.005107:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3688 00000100:00080000:1.0:1787830682.005113:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830682.005119:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:3.0:1787830682.005122:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:1.0:1787830682.005124:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:3.0:1787830682.005129:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830682.005135:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518306432:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830682.005139:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518306944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830682.005148:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830682.005151:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830682.005153:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830682.005156:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3688 00000100:00080000:1.0:1787830682.005161:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830682.005161:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518306560:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830682.005168:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830682.005172:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830682.005183:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518307072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830682.005193:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:3.0:1787830682.005196:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830682.005199:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830682.005203:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830682.005208:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3688 00000100:00080000:1.0:1787830682.005211:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830682.005216:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830682/real 1787830682] req@ffff8b2e88075e00 x1874672518306560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830682.005217:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830682.005224:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830682.005238:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518307200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830682.005240:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518306560/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830682.005266:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830682.005271:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830682.005275:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518306560:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830682.005283:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518306688:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830682.005304:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830682.005316:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830682/real 1787830682] req@ffff8b2e88074b40 x1874672518306688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830682.005327:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518306688/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830682.005333:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830682.005336:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830682.005339:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518306688:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830682.005343:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518306816:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830682.005359:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830682.005383:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830682/real 1787830682] req@ffff8b2e88074000 x1874672518306816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830682.005394:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518306816/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830682.005402:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830682.005405:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830682.005409:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518306816:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830682.005414:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518306944:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830682.005432:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830682.005444:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830682/real 1787830682] req@ffff8b2e880761c0 x1874672518306944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830682.005455:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518306944/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830682.005462:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830682.005465:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830682.005468:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518306944:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830682.005485:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518307072:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830682.005504:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830682.005514:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830682/real 1787830682] req@ffff8b2e880770c0 x1874672518307072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830682.005525:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518307072/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830682.005534:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830682.005537:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830682.005540:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518307072:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830682.005545:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518307200:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830682.005563:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830682.005574:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830682/real 1787830682] req@ffff8b2e88077840 x1874672518307200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830682.005584:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518307200/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830707 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830682.005591:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830682.005594:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830682.005597:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518307200:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830682.007678:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830682.325306:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830682.836841:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830682.836913:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830683.028787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830683.030753:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830683.350223:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830683.860897:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830683.862404:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830684.052781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830684.053441:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830684.372803:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830684.884975:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830684.885531:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830685.076787:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830685.078186:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830685.396801:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830685.908790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830685.909141:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830686.100827:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830686.100901:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830686.420795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830686.932851:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830686.932927:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830687.124824:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830687.124927:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830687.124939:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830687.124945:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830687.124958:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3693 00000100:00080000:1.0:1787830687.124979:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830687.124990:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830687.125039:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518307328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830687.125124:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518307328:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830687.125127:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000800:00000100:3.0:1787830687.125195:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830687.125226:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830687.125230:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830687.125231:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830687.125235:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3693 00000100:00080000:1.0:1787830687.125238:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830687.125241:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830687.125250:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830687.125263:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518307456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830687.125381:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830687/real 1787830687] req@ffff8b2e88077480 x1874672518307328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830687.125400:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518307328/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830687.125412:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830687.125416:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830687.125420:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518307328:192.168.203.101@tcp:250:kworker.0 00000100:00080000:1.0:1787830687.125440:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830687.125445:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830687.125446:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830687.125451:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3693 00000100:00080000:1.0:1787830687.125455:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830687.125460:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830687.125466:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830687.125479:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518307584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830687.125488:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830687.125491:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830687.125493:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830687.125496:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3693 00000100:00080000:1.0:1787830687.125499:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830687.125503:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830687.125507:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830687.125511:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518307456:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830687.125517:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518307712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830687.125526:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830687.125529:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830687.125531:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830687.125535:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3693 00000800:00000100:3.0:1787830687.125536:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830687.125538:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830687.125542:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830687.125546:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830687.125548:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830687/real 1787830687] req@ffff8b2e88075680 x1874672518307456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830687.125555:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518307840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830687.125560:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518307456/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830687.125563:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830687.125567:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830687.125568:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830687.125569:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830687.125572:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3693 00000100:00080000:3.0:1787830687.125572:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:1.0:1787830687.125575:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:3.0:1787830687.125576:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518307456:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830687.125579:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830687.125581:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518307584:192.168.203.101@tcp:8:kworker.0 00000008:00100000:1.0:1787830687.125583:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830687.125591:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518307968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:3.0:1787830687.125598:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:1.0:1787830687.125606:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:3.0:1787830687.125607:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830687/real 1787830687] req@ffff8b2e88074780 x1874672518307584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830687.125609:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830687.125611:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830687.125615:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3693 00000100:00080000:1.0:1787830687.125618:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830687.125633:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830687.125636:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518307584/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:1.0:1787830687.125638:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830687.125951:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518308096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830687.126307:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830687.126314:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830687.126319:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518307584:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830687.126346:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518307712:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830687.126378:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830687.126393:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830687/real 1787830687] req@ffff8b2e88074f00 x1874672518307712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830687.126407:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518307712/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830687.126415:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830687.126419:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830687.126422:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518307712:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830687.126578:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518307840:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830687.126606:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830687.126618:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830687/real 1787830687] req@ffff8b2e88075a40 x1874672518307840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830687.126643:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518307840/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830687.126681:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830687.126684:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830687.126688:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518307840:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830687.126694:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518307968:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830687.126713:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830687.126724:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830687/real 1787830687] req@ffff8b2e88076d00 x1874672518307968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830687.126734:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518307968/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830687.126742:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830687.126745:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830687.126748:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518307968:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830687.126764:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518308096:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830687.127061:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830687.127075:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830687/real 1787830687] req@ffff8b2e88077c00 x1874672518308096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830687.127089:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518308096/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830712 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830687.127097:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830687.127100:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830687.127104:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518308096:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830687.445580:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830687.956960:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830687.957065:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830688.148824:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830688.148996:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830688.468907:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830688.980863:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830688.980894:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830689.172798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830689.173962:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830689.492788:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830690.004884:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830690.005165:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830690.196798:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830690.197177:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830690.516806:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830691.028857:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830691.028932:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830691.220805:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830691.220898:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830691.540775:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830692.052782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830692.053572:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830692.244896:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830692.244908:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830692.244912:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830692.244924:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3698 00000100:00080000:3.0:1787830692.244938:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830692.244946:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830692.244992:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518308224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830692.245027:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830692.245031:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830692.245033:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830692.245037:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3698 00000100:00080000:3.0:1787830692.245041:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830692.245044:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830692.245057:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830692.245072:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518308352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830692.245081:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830692.245085:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830692.245086:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830692.245091:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3698 00000100:00080000:3.0:1787830692.245094:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830692.245104:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830692.245105:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518308224:192.168.203.101@tcp:250:kworker.0 00000008:00100000:3.0:1787830692.245109:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830692.245131:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518308480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830692.245144:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830692.245147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830692.245149:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830692.245153:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3698 00000100:00080000:3.0:1787830692.245156:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830692.245160:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830692.245166:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830692.245177:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518308608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830692.245185:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830692.245192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000800:00000100:1.0:1787830692.245194:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830692.245196:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830692.245202:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3698 00000100:00080000:3.0:1787830692.245207:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830692.245212:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830692.245218:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830692.245230:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518308736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830692.245238:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830692.245242:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830692.245244:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830692.245248:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3698 00000100:00080000:3.0:1787830692.245253:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830692.245257:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830692.245263:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830692.245265:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830692/real 1787830692] req@ffff8b2e88039680 x1874672518308224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830692.245277:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518308864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830692.245288:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830692.245303:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830692.245308:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830692.245309:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518308224/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830692.245314:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3698 00000100:00080000:3.0:1787830692.245321:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830692.245329:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:1.0:1787830692.245333:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787830692.245336:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787830692.245343:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830692.245349:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518308224:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830692.245354:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518308992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830692.245367:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518308352:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830692.245402:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830692.245422:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830692/real 1787830692] req@ffff8b2e88039a40 x1874672518308352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830692.245434:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518308352/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830692.245443:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830692.245447:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830692.245452:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518308352:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830692.245454:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830692.245485:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518308480:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830692.245513:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830692.245533:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830692/real 1787830692] req@ffff8b2e8803a580 x1874672518308480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830692.245558:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518308480/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830692.245571:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830692.245576:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830692.245580:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518308480:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830692.245588:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518308608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830692.245598:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000800:00000100:1.0:1787830692.245613:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830692.245629:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830692/real 1787830692] req@ffff8b2e8803a1c0 x1874672518308608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830692.245639:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518308608/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830692.245672:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830692.245676:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830692.245679:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518308608:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830692.245700:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518308736:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830692.245720:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830692.245730:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830692/real 1787830692] req@ffff8b2e880392c0 x1874672518308736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830692.245740:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518308736/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830692.245748:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830692.245751:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830692.245754:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518308736:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830692.245760:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518308864:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830692.245778:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830692.245788:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830692/real 1787830692] req@ffff8b2e8803b480 x1874672518308864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830692.245799:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518308864/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830692.245807:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830692.245810:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830692.245813:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518308864:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830692.245819:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518308992:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830692.245836:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830692.245846:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830692/real 1787830692] req@ffff8b2e88039e00 x1874672518308992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830692.245854:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518308992/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830717 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830692.245861:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830692.245863:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830692.245866:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518308992:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830692.565311:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830693.076804:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830693.076881:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830693.269276:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830693.270084:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830693.590292:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830694.100867:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830694.100900:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830694.292782:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830694.292927:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830694.613185:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830695.125021:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830695.125417:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830695.316881:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830695.317851:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830695.636902:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830696.148931:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830696.149200:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830696.341354:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830696.341534:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830696.660951:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830697.172785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830697.173024:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:3.0:1787830697.364892:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830697.364895:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830697.364904:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830697.364908:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830697.364918:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3703 00000100:00080000:3.0:1787830697.364928:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830697.364935:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830697.364972:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518309120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830697.365054:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518309120:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830697.365109:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830697.365147:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830697.365151:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830697.365153:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830697.365157:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3703 00000100:00080000:3.0:1787830697.365161:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830697.365166:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830697.365175:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830697.365187:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518309248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830697.366594:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830697.366623:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830697/real 1787830697] req@ffff8b2e8803b0c0 x1874672518309120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830697.366632:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830697.366639:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830697.366642:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830697.366667:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3703 00000100:00080000:3.0:1787830697.366674:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830697.366675:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518309120/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830697.366682:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00080000:1.0:1787830697.366693:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000008:00100000:3.0:1787830697.366694:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00080000:1.0:1787830697.366701:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830697.366706:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518309120:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830697.366717:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518309376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830697.366729:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830697.366732:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830697.366734:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830697.366738:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3703 00000100:00080000:3.0:1787830697.366742:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830697.366746:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518309248:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830697.366749:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830697.366758:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830697.366773:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518309504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830697.366784:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000800:00000100:1.0:1787830697.366791:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830697.366802:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830697.366805:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830697.366809:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830697/real 1787830697] req@ffff8b2e88038f00 x1874672518309248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830697.366816:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3703 00000100:00080000:3.0:1787830697.366824:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830697.366829:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518309248/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830697.366831:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830697.366840:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:1.0:1787830697.366843:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830697.366848:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830697.366852:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518309248:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830697.366858:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518309632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830697.366861:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518309376:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830697.366876:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830697.366879:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830697.366881:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830697.366900:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3703 00000800:00000100:1.0:1787830697.366903:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:3.0:1787830697.366907:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830697.366913:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830697.366919:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830697.366920:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830697/real 1787830697] req@ffff8b2e880383c0 x1874672518309376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830697.366935:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518309376/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830697.366945:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518309760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830697.366956:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830697.366957:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830697.366964:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830697.366965:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787830697.366967:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:1.0:1787830697.366973:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518309376:192.168.203.101@tcp:8:kworker.0 00000100:00080000:3.0:1787830697.366976:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3703 00000100:00080000:3.0:1787830697.366983:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830697.366987:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830697.366992:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830697.367004:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518309888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830697.367008:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518309504:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830697.367042:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830697.367055:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830697/real 1787830697] req@ffff8b2e8803b840 x1874672518309504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830697.367067:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518309504/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830697.367075:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830697.367078:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830697.367081:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518309504:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830697.367086:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518309632:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830697.367104:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830697.367114:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830697/real 1787830697] req@ffff8b2e8803b0c0 x1874672518309632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830697.367123:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518309632/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830697.367130:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830697.367132:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830697.367135:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518309632:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830697.367141:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518309760:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830697.367158:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830697.367168:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830697/real 1787830697] req@ffff8b2e88039e00 x1874672518309760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830697.367185:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518309760/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830697.367202:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830697.367205:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830697.367208:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518309760:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830697.367222:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518309888:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830697.367240:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830697.367250:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830697/real 1787830697] req@ffff8b2e8803b480 x1874672518309888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830697.367278:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518309888/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830722 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830697.367286:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830697.367289:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830697.367293:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518309888:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830697.684795:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830698.196774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830698.199060:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830698.388764:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830698.388764:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830698.708827:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830699.220723:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830699.220875:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830699.412836:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830699.412836:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830699.734183:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830700.244778:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830700.246030:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830700.436775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830700.437256:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830700.756785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830701.270338:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830701.270363:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830701.460810:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830701.460948:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830701.780790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830702.292781:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830702.292822:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830702.484780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830702.484780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830702.484825:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830702.484831:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830702.484834:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830702.484842:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3708 00000100:00080000:0.0:1787830702.484854:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830702.484860:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830702.484897:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880403c0 x1874672518310016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830702.484920:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830702.484923:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830702.484932:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830702.484937:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3708 00000100:00080000:0.0:1787830702.484940:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830702.484944:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830702.484954:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830702.484968:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880430c0 x1874672518310144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830702.484976:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518310016:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787830702.484981:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830702.484985:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830702.484987:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830702.484991:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3708 00000100:00080000:0.0:1787830702.484994:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830702.484998:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830702.485002:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830702.485014:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040000 x1874672518310272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830702.485021:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830702.485025:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830702.485026:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830702.485029:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3708 00000100:00080000:0.0:1787830702.485033:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:1.0:1787830702.485037:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830702.485038:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830702.485045:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830702.485056:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042940 x1874672518310400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830702.485064:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830702.485067:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830702.485069:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830702.485072:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3708 00000100:00080000:0.0:1787830702.485075:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830702.485078:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830702.485082:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830702.485091:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043480 x1874672518310528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830702.485099:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:1.0:1787830702.485102:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830702/real 1787830702] req@ffff8b2e880403c0 x1874672518310016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830702.485104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830702.485107:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830702.485114:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3708 00000100:00080000:0.0:1787830702.485120:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830702.485124:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830702.485128:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830702.485128:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880403c0 x1874672518310016/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830702.485140:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787830702.485142:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041a40 x1874672518310656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830702.485145:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830702.485152:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880403c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518310016:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787830702.485160:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830702.485164:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830702.485165:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830702.485169:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3708 00000100:00080000:0.0:1787830702.485172:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00100000:1.0:1787830702.485175:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518310144:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787830702.485177:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830702.485183:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830702.485195:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040f00 x1874672518310784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:1.0:1787830702.485202:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830702.485234:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830702/real 1787830702] req@ffff8b2e880430c0 x1874672518310144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830702.485248:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880430c0 x1874672518310144/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830702.485256:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830702.485259:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830702.485262:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880430c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518310144:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830702.485268:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518310272:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830702.485296:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830702.485308:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830702/real 1787830702] req@ffff8b2e88040000 x1874672518310272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830702.485325:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040000 x1874672518310272/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830702.485344:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830702.485346:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830702.485348:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518310272:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830702.485352:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518310400:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830702.485368:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830702.485378:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830702/real 1787830702] req@ffff8b2e88042940 x1874672518310400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830702.485389:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042940 x1874672518310400/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830702.485396:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830702.485398:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830702.485401:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518310400:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830702.485406:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518310528:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830702.485422:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830702.485432:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830702/real 1787830702] req@ffff8b2e88043480 x1874672518310528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830702.485442:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043480 x1874672518310528/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830702.485453:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830702.485456:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830702.485458:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518310528:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830702.485463:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518310656:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830702.485479:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830702.485488:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830702/real 1787830702] req@ffff8b2e88041a40 x1874672518310656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830702.485497:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041a40 x1874672518310656/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830702.485504:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830702.485507:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830702.485509:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518310656:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830702.485525:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518310784:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830702.485542:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830702.485552:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830702/real 1787830702] req@ffff8b2e88040f00 x1874672518310784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830702.485561:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040f00 x1874672518310784/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830727 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830702.485568:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830702.485570:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830702.485574:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518310784:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830702.804786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830703.316777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830703.316812:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830703.508793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830703.508806:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830703.828796:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830704.340783:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830704.342022:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830704.532770:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830704.532770:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830704.853274:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830705.364964:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830705.365934:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830705.556793:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830705.556889:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830705.877387:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830706.388761:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830706.388847:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830706.580909:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830706.580962:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830706.900792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830707.412944:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830707.413014:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830707.605108:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:2.0:1787830707.605207:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830707.605213:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:2.0:1787830707.605216:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830707.605225:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3714 00000100:00080000:2.0:1787830707.605235:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830707.605242:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:2.0:1787830707.605278:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801ed00 x1874672518310912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830707.605306:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830707.605310:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830707.605311:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830707.605314:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3714 00000100:00080000:2.0:1787830707.605317:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830707.605321:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830707.605330:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830707.605342:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c780 x1874672518311040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830707.605350:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830707.605353:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830707.605355:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830707.605357:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3714 00000100:00080000:2.0:1787830707.605359:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830707.605362:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830707.605366:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830707.605373:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801f0c0 x1874672518311168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830707.605381:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830707.605384:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830707.605385:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830707.605387:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3714 00000100:00080000:2.0:1787830707.605389:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830707.605392:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830707.605395:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830707.605401:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801d680 x1874672518311296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830707.605407:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830707.605409:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:2.0:1787830707.605411:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830707.605413:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3714 00000100:00080000:2.0:1787830707.605415:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830707.605418:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830707.605421:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830707.605427:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801c3c0 x1874672518311424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830707.605435:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830707.605437:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:2.0:1787830707.605438:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830707.605441:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3714 00000100:00080000:2.0:1787830707.605444:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830707.605447:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830707.605450:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830707.605456:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801cb40 x1874672518311552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830707.605463:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830707.605465:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830707.605466:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830707.605469:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3714 00000100:00080000:2.0:1787830707.605471:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:2.0:1787830707.605473:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:2.0:1787830707.605476:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830707.605482:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8801de00 x1874672518311680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830707.605721:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518310912:192.168.203.101@tcp:250:kworker.0 00000800:00000100:1.0:1787830707.605773:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830707.605812:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830707/real 1787830707] req@ffff8b2e8801ed00 x1874672518310912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830707.605827:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801ed00 x1874672518310912/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830707.605837:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830707.605841:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830707.605844:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801ed00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518310912:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830707.605850:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518311040:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830707.605870:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830707.605881:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830707/real 1787830707] req@ffff8b2e8801c780 x1874672518311040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830707.605893:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c780 x1874672518311040/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830707.605901:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830707.605904:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830707.605907:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518311040:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830707.605914:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518311168:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830707.605931:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830707.605941:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830707/real 1787830707] req@ffff8b2e8801f0c0 x1874672518311168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830707.605950:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801f0c0 x1874672518311168/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830707.605957:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830707.605960:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830707.605963:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801f0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518311168:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830707.605968:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518311296:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830707.605984:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830707.605993:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830707/real 1787830707] req@ffff8b2e8801d680 x1874672518311296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830707.606002:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801d680 x1874672518311296/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830707.606008:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830707.606010:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830707.606013:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801d680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518311296:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830707.606018:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518311424:192.168.203.101@tcp:38:kworker.0 00000800:00000100:1.0:1787830707.606034:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830707.606044:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830707/real 1787830707] req@ffff8b2e8801c3c0 x1874672518311424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830707.606052:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801c3c0 x1874672518311424/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830707.606060:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830707.606062:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830707.606065:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801c3c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518311424:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830707.606070:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518311552:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830707.606086:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830707.606095:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830707/real 1787830707] req@ffff8b2e8801cb40 x1874672518311552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830707.606103:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801cb40 x1874672518311552/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830707.606110:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830707.606113:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830707.606116:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801cb40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518311552:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830707.606121:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518311680:192.168.203.101@tcp:8:kworker.0 00000800:00000100:1.0:1787830707.606138:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:1.0:1787830707.606148:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830707/real 1787830707] req@ffff8b2e8801de00 x1874672518311680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830707.606177:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8801de00 x1874672518311680/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830732 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830707.606194:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830707.606197:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830707.606201:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8801de00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518311680:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830707.606246:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830707.924790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830708.436816:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830708.437293:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830708.628843:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830708.630802:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830708.948789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830709.460776:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830709.464288:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830709.652834:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830709.652940:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830709.973159:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830710.484794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830710.484954:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830710.676749:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830710.677609:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830710.996800:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830711.508768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830711.509208:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830711.701203:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830711.704297:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830712.020792:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830712.532760:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830712.533216:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830712.724817:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830712.724890:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830712.726123:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830712.726131:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830712.726136:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830712.726145:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3719 00000100:00080000:1.0:1787830712.726165:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830712.726172:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830712.726209:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518311808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830712.726241:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830712.726246:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830712.726248:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830712.726251:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3719 00000100:00080000:1.0:1787830712.726255:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830712.726259:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830712.726269:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830712.726283:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518311936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830712.726292:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830712.726295:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830712.726297:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830712.726300:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3719 00000100:00080000:1.0:1787830712.726303:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830712.726307:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830712.726311:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830712.726321:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518312064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830712.726328:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830712.726331:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830712.726332:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830712.726335:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3719 00000100:00080000:1.0:1787830712.726338:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830712.726342:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830712.726346:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830712.726355:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518312192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830712.726364:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830712.726367:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830712.726369:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830712.726372:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3719 00000100:00080000:1.0:1787830712.726375:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830712.726379:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830712.726383:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830712.726392:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518312320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830712.726401:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830712.726404:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830712.726406:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830712.726409:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3719 00000100:00080000:1.0:1787830712.726412:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830712.726415:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830712.726419:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830712.726428:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518312448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830712.726435:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830712.726438:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830712.726439:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830712.726442:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3719 00000100:00080000:1.0:1787830712.726445:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830712.726448:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830712.726452:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830712.726460:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518312576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830712.726673:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518311808:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830712.726729:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830712.726971:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830712/real 1787830712] req@ffff8b2e88076940 x1874672518311808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830712.726983:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518311808/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830712.726991:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830712.726993:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830712.726996:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518311808:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830712.727001:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518311936:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830712.727018:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830712.727026:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830712/real 1787830712] req@ffff8b2e88077c00 x1874672518311936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830712.727033:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518311936/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830712.727038:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830712.727040:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830712.727042:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518311936:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830712.727045:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518312064:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830712.727055:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830712.727060:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830712/real 1787830712] req@ffff8b2e88076d00 x1874672518312064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830712.727066:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518312064/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830712.727075:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830712.727076:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830712.727078:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518312064:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830712.727082:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518312192:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830712.727096:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830712.727105:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830712/real 1787830712] req@ffff8b2e88075a40 x1874672518312192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830712.727115:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518312192/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830712.727123:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830712.727126:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830712.727129:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518312192:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830712.727135:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518312320:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830712.727165:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830712.727182:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830712/real 1787830712] req@ffff8b2e88074f00 x1874672518312320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830712.727193:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518312320/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830712.727201:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830712.727204:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830712.727207:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518312320:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830712.727213:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518312448:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830712.727232:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830712.727248:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830712/real 1787830712] req@ffff8b2e88074780 x1874672518312448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830712.727259:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518312448/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830712.727266:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830712.727269:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830712.727273:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518312448:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830712.727278:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518312576:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830712.727294:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830712.727301:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830712/real 1787830712] req@ffff8b2e88075680 x1874672518312576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830712.727308:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518312576/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830737 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830712.727313:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830712.727315:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830712.727316:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518312576:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830713.044932:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:3.0:1787830713.556779:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830713.557204:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830713.749053:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830713.749511:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830714.069387:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:2.0:1787830714.581914:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830714.582561:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830714.772780:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830714.772780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830715.092786:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830715.607195:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830715.607868:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830715.796860:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830715.797189:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830716.116783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830716.629101:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830716.629513:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830716.820786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830716.822932:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830717.140839:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830717.652793:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830717.655330:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830717.844861:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830717.844872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830717.844876:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830717.844886:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3724 00000100:00080000:1.0:1787830717.844897:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830717.844904:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830717.845076:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518312704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830717.845103:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830717.845108:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830717.845110:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830717.845129:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3724 00000100:00080000:1.0:1787830717.845134:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830717.845139:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830717.845150:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830717.845162:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518312832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830717.845171:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830717.845174:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830717.845175:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830717.845179:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3724 00000100:00080000:1.0:1787830717.845182:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830717.845185:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830717.845189:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830717.845198:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518312960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830717.845205:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830717.845208:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830717.845210:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830717.845213:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3724 00000100:00080000:1.0:1787830717.845216:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830717.845219:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830717.845223:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830717.845232:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518313088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830717.845240:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830717.845242:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830717.845244:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830717.845247:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3724 00000100:00080000:1.0:1787830717.845250:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830717.845253:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830717.845257:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830717.845266:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518313216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830717.845287:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830717.845290:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830717.845292:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830717.845294:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3724 00000100:00080000:1.0:1787830717.845297:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830717.845300:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830717.845304:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830717.845314:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518313344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830717.845322:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830717.845325:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830717.845327:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830717.845330:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3724 00000100:00080000:1.0:1787830717.845332:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830717.845336:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830717.845340:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830717.845350:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518313472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830717.845502:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830717.849579:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518312704:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830717.850091:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830717.850152:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830717/real 1787830717] req@ffff8b2e88077480 x1874672518312704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830717.850172:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518312704/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830717.850183:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830717.850187:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830717.850191:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518312704:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830717.850199:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518312832:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830717.850224:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830717.850237:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830717/real 1787830717] req@ffff8b2e88077840 x1874672518312832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830717.850248:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518312832/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830717.850257:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830717.850260:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830717.850263:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518312832:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830717.850270:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518312960:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830717.850292:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830717.850304:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830717/real 1787830717] req@ffff8b2e880770c0 x1874672518312960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830717.850315:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518312960/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830717.850324:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830717.850327:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830717.850330:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518312960:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830717.850335:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518313088:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830717.850355:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830717.850366:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830717/real 1787830717] req@ffff8b2e880761c0 x1874672518313088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830717.850378:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518313088/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830717.850385:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830717.850388:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830717.850391:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518313088:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830717.850397:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518313216:192.168.203.101@tcp:38:kworker.0 00000800:00000100:3.0:1787830717.850416:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830717.850428:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830717/real 1787830717] req@ffff8b2e88074000 x1874672518313216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830717.850445:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518313216/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830717.850453:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830717.850456:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830717.850459:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518313216:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830717.850464:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518313344:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830717.850483:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830717.850494:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830717/real 1787830717] req@ffff8b2e88074b40 x1874672518313344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830717.850780:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518313344/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830717.850789:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830717.850792:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830717.850795:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518313344:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830717.850812:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518313472:192.168.203.101@tcp:8:kworker.0 00000800:00000100:3.0:1787830717.850836:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830717.851033:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830717/real 1787830717] req@ffff8b2e88075e00 x1874672518313472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:3.0:1787830717.851046:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518313472/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830742 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830717.851060:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830717.851063:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:3.0:1787830717.851067:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518313472:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830717.851213:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830718.165275:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830718.677581:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830718.679210:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830718.868786:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830718.868786:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830719.188789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830719.700790:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830719.700989:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830719.893406:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830719.895166:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830720.212730:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830720.725912:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830720.729607:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830720.916792:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830720.917977:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830721.236882:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830721.750991:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830721.751398:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830721.940776:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830721.941159:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830722.261202:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830722.772983:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:1.0:1787830722.774932:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830722.964815:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830722.964828:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830722.964832:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830722.964843:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3729 00000100:00080000:1.0:1787830722.964855:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830722.964862:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830722.964900:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076580 x1874672518313600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830722.964927:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830722.964931:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830722.964932:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830722.964936:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3729 00000100:00080000:1.0:1787830722.964940:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830722.964943:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830722.964954:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830722.964969:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518313728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830722.964978:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830722.964981:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830722.964984:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830722.964987:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3729 00000100:00080000:1.0:1787830722.964990:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830722.964993:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830722.964998:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830722.965008:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518313856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830722.965017:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830722.965020:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830722.965022:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830722.965025:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3729 00000100:00080000:1.0:1787830722.965028:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830722.965031:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830722.965035:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830722.965044:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518313984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830722.965054:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830722.965056:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830722.965058:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830722.965062:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3729 00000100:00080000:1.0:1787830722.965065:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830722.965068:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830722.965072:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830722.965093:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518314112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830722.965102:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830722.965105:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830722.965106:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830722.965109:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3729 00000100:00080000:1.0:1787830722.965112:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830722.965115:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830722.965119:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830722.965127:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518314240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830722.965135:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830722.965138:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830722.965139:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830722.965142:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3729 00000100:00080000:1.0:1787830722.965145:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830722.965148:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830722.965152:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830722.965162:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518314368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830722.965268:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830722.965361:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518313600:192.168.203.101@tcp:250:kworker.0 00000100:00100000:3.0:1787830722.965473:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518313728:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830722.965500:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518313856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830722.965520:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518313984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830722.965537:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518314112:192.168.203.101@tcp:38:kworker.0 00000100:00100000:3.0:1787830722.965554:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518314240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830722.965571:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518314368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830722.969188:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830723.284804:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830723.796787:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830723.797098:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830723.988780:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830723.992140:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830724.309074:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830724.820969:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830724.821596:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830725.012760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830725.012760:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830725.332783:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830725.844820:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830725.845765:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830726.036864:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830726.038661:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000400:00000100:3.0:1787830726.041515:0:2336:0:(lib-socket.c:438:lnet_sock_connect()) Error -113 connecting 1023 -> 192.168.203.101:988 00000400:00000100:3.0:1787830726.041572:0:2336:0:(acceptor.c:117:lnet_connect_console_error()) Connection to 192.168.203.101@tcp at host 192.168.203.101 was unreachable: the network or that node may be down, or Lustre may be misconfigured. 00000800:00000100:3.0:1787830726.041588:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:1.0:1787830726.045324:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830722/real 1787830726] req@ffff8b2e88076580 x1874672518313600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830747 ref 2 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830726.045359:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076580 x1874672518313600/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830747 ref 2 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830726.045368:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830726.045371:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830726.045374:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518313600:192.168.203.101@tcp:250:kworker.0 00000800:00000100:3.0:1787830726.045530:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830726.045559:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830726.045567:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830726.045571:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830726.045576:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000800:00000100:3.0:1787830726.045583:0:2336:0:(socklnd_cb.c:353:ksocknal_txlist_done()) Deleting packet type 1 len 520 0@<0:0>->192.168.203.101@tcp 00000100:00100000:1.0:1787830726.045629:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830722/real 1787830726] req@ffff8b2e88075e00 x1874672518313728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830726.045714:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518313728/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830726.045813:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830726.045818:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830726.045822:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518313728:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830726.045828:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830722/real 1787830726] req@ffff8b2e88074b40 x1874672518313856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830726.045840:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518313856/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830726.045847:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830726.045850:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830726.045853:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518313856:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830726.045858:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830722/real 1787830726] req@ffff8b2e88074000 x1874672518313984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830726.045869:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518313984/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830726.045876:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830726.045880:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830726.045883:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518313984:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830726.045888:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830722/real 1787830726] req@ffff8b2e880761c0 x1874672518314112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830726.045899:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518314112/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830726.045906:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830726.045909:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830726.045912:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518314112:192.168.203.101@tcp:38:kworker.0 00000100:00100000:1.0:1787830726.045918:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830722/real 1787830726] req@ffff8b2e880770c0 x1874672518314240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830726.045928:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518314240/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830726.045936:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830726.045939:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830726.045942:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518314240:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830726.045949:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830722/real 1787830726] req@ffff8b2e88077840 x1874672518314368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830747 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:1.0:1787830726.045959:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518314368/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830747 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:1.0:1787830726.045966:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:1.0:1787830726.045969:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:1.0:1787830726.045972:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518314368:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830726.356789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830726.868730:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:2.0:1787830726.869122:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830727.060772:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830727.060833:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830727.380782:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830727.892806:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830727.893466:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830728.084823:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830728.084830:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830728.084835:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830728.084837:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830728.084842:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830728.084853:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3734 00000100:00080000:3.0:1787830728.084925:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830728.084934:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830728.084965:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518314496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830728.084985:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830728.084988:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830728.084990:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830728.084994:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3734 00000100:00080000:3.0:1787830728.084997:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830728.085000:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830728.085009:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830728.085019:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518314624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830728.085028:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830728.085031:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830728.085033:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830728.085037:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3734 00000100:00080000:3.0:1787830728.085052:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00100000:0.0:1787830728.085057:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518314496:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830728.085059:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830728.085068:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830728.085081:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518314752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830728.085091:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830728.085094:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830728.085096:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830728.085100:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3734 00000100:00080000:3.0:1787830728.085104:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830728.085107:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830728.085112:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000800:00000100:0.0:1787830728.085122:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:3.0:1787830728.085124:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518314880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830728.085132:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830728.085135:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830728.085138:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830728.085141:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3734 00000100:00080000:3.0:1787830728.085144:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830728.085147:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830728.085151:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830728.085163:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518315008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830728.085166:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830728/real 1787830728] req@ffff8b2e880392c0 x1874672518314496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830728.085179:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830728.085186:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830728.085188:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830728.085193:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3734 00000100:00080000:3.0:1787830728.085197:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830728.085203:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830728.085204:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518314496/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000008:00100000:3.0:1787830728.085209:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:0.0:1787830728.085222:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:3.0:1787830728.085225:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518315136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830728.085229:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830728.085238:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518314496:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830728.085242:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830728.085247:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00100000:0.0:1787830728.085250:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518314624:192.168.203.101@tcp:38:kworker.0 00000100:00080000:3.0:1787830728.085251:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830728.085259:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3734 00000100:00080000:3.0:1787830728.085265:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830728.085269:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830728.085273:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830728.085284:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518315264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:0.0:1787830728.085288:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830728.085306:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830728/real 1787830728] req@ffff8b2e8803a1c0 x1874672518314624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830728.085321:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518314624/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830728.085329:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830728.085332:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830728.085335:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518314624:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830728.085355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518314752:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830728.085374:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830728.085384:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830728/real 1787830728] req@ffff8b2e8803a580 x1874672518314752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830728.085393:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518314752/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830728.085401:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830728.085403:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830728.085406:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518314752:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830728.085411:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518314880:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830728.085426:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830728.085434:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830728/real 1787830728] req@ffff8b2e88039a40 x1874672518314880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830728.085451:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518314880/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830728.085458:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830728.085461:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830728.085463:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518314880:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830728.085468:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518315008:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830728.085484:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830728.085493:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830728/real 1787830728] req@ffff8b2e88039680 x1874672518315008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830728.085504:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518315008/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830728.085512:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830728.085515:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830728.085518:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518315008:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830728.085524:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518315136:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830728.085541:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830728.085557:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830728/real 1787830728] req@ffff8b2e8803b480 x1874672518315136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830728.085568:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518315136/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830728.085581:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830728.085584:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830728.085587:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518315136:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830728.085592:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518315264:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830728.085610:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830728.085621:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830728/real 1787830728] req@ffff8b2e88039e00 x1874672518315264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830728.085632:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518315264/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830753 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830728.085640:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830728.085728:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830728.085732:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518315264:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830728.404899:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830728.916785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830728.916864:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830729.108781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830729.108872:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830729.428790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830729.940957:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830729.941117:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830730.132781:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830730.132781:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830730.452793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830730.964791:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830730.964961:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830731.156905:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830731.157178:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830731.476852:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830731.988807:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830731.988837:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830732.180807:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830732.180850:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830732.502287:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830733.012941:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830733.015507:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830733.204802:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830733.205244:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830733.205250:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830733.205254:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830733.205264:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3739 00000100:00080000:0.0:1787830733.205273:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830733.205281:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830733.205344:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880421c0 x1874672518315392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830733.205378:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830733.205382:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830733.205384:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830733.205388:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3739 00000100:00080000:0.0:1787830733.205392:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830733.205396:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830733.205405:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830733.205418:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518315520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830733.205428:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830733.205431:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830733.205433:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830733.205437:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3739 00000100:00080000:0.0:1787830733.205441:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830733.205447:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830733.205452:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:2.0:1787830733.205453:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518315392:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830733.205465:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518315648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830733.205475:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830733.205479:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830733.205481:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830733.205484:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3739 00000100:00080000:0.0:1787830733.205488:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830733.205492:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830733.205497:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830733.205509:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518315776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830733.205512:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830733.205523:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830733.205526:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830733.205528:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830733.205534:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3739 00000100:00080000:0.0:1787830733.205538:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830733.205542:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830733.205546:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830733.205557:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518315904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830733.205565:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:2.0:1787830733.205569:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830733/real 1787830733] req@ffff8b2e880421c0 x1874672518315392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830733.205570:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830733.205573:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830733.205579:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3739 00000100:00080000:0.0:1787830733.205583:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830733.205587:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830733.205594:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830733.205606:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518316032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830733.205614:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880421c0 x1874672518315392/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830733.205635:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:2.0:1787830733.205638:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830733.205640:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830733.205644:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830733.205668:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830733.205673:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880421c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518315392:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830733.205679:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518315520:192.168.203.101@tcp:38:kworker.0 00000100:00080000:0.0:1787830733.205684:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3739 00000100:00080000:0.0:1787830733.205689:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830733.205693:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830733.205698:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830733.205710:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518316160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000800:00000100:2.0:1787830733.205719:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830733.205738:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830733/real 1787830733] req@ffff8b2e88040b40 x1874672518315520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830733.205762:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518315520/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830733.205771:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830733.205775:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830733.205778:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518315520:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830733.205813:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518315648:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830733.205834:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830733.205853:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830733/real 1787830733] req@ffff8b2e880412c0 x1874672518315648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830733.205865:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518315648/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830733.205872:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830733.205876:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830733.205879:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518315648:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830733.205884:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518315776:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830733.205910:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830733.205922:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830733/real 1787830733] req@ffff8b2e88043840 x1874672518315776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830733.205937:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518315776/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830733.205951:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830733.205954:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830733.205957:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518315776:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830733.205963:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518315904:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830733.205982:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830733.205993:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830733/real 1787830733] req@ffff8b2e88040780 x1874672518315904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830733.206025:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518315904/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830733.206033:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830733.206037:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830733.206040:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518315904:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830733.206045:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518316032:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830733.206068:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830733.206076:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830733/real 1787830733] req@ffff8b2e88042d00 x1874672518316032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830733.206083:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518316032/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830733.206088:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830733.206090:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830733.206092:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518316032:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830733.206095:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518316160:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830733.206109:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830733.206115:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830733/real 1787830733] req@ffff8b2e88042580 x1874672518316160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830733.206122:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518316160/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830758 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830733.206127:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830733.206129:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830733.206130:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518316160:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830733.207926:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830733.524816:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830734.037166:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830734.038245:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:1.0:1787830734.228797:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830734.228910:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830734.548723:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830735.060764:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830735.061077:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830735.252836:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830735.254487:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830735.572785:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830736.084785:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830736.085040:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830736.276714:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830736.276729:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830736.596809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830737.109468:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830737.109483:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830737.300753:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830737.300938:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830737.620773:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830738.132849:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830738.132955:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830738.324735:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:0.0:1787830738.324792:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830738.324798:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:0.0:1787830738.324802:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830738.324811:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3744 00000100:00080000:0.0:1787830738.324822:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830738.324829:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:0.0:1787830738.324866:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88041e00 x1874672518316288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:1.0:1787830738.324872:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:0.0:1787830738.324897:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830738.324901:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830738.324903:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830738.324907:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3744 00000100:00080000:0.0:1787830738.324910:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830738.324914:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830738.324924:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830738.324939:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042580 x1874672518316416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830738.324945:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518316288:192.168.203.101@tcp:250:kworker.0 00000100:00080000:0.0:1787830738.324956:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830738.324960:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830738.324962:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830738.324966:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3744 00000100:00080000:0.0:1787830738.324981:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830738.324985:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830738.324991:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:0.0:1787830738.325004:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88042d00 x1874672518316544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830738.325012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830738.325015:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830738.325016:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830738.325018:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3744 00000100:00080000:0.0:1787830738.325021:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787830738.325025:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00080000:0.0:1787830738.325027:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830738.325033:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830738.325040:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040780 x1874672518316672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830738.325046:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830738.325048:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:0.0:1787830738.325050:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830738.325052:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3744 00000100:00080000:0.0:1787830738.325054:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830738.325056:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830738.325059:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830738.325065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88043840 x1874672518316800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830738.325070:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830738.325072:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:0.0:1787830738.325074:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:0.0:1787830738.325078:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3744 00000100:00100000:2.0:1787830738.325078:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830738/real 1787830738] req@ffff8b2e88041e00 x1874672518316288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830738.325079:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830738.325084:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830738.325089:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:0.0:1787830738.325098:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880412c0 x1874672518316928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:0.0:1787830738.325103:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:0.0:1787830738.325106:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:0.0:1787830738.325108:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830738.325108:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88041e00 x1874672518316288/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830738.325114:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3744 00000100:00080000:0.0:1787830738.325118:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:0.0:1787830738.325123:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:0.0:1787830738.325128:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00080000:2.0:1787830738.325130:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00100000:0.0:1787830738.325135:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88040b40 x1874672518317056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:2.0:1787830738.325137:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830738.325147:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88041e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518316288:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830738.325173:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518316416:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830738.325198:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830738.325211:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830738/real 1787830738] req@ffff8b2e88042580 x1874672518316416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830738.325224:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042580 x1874672518316416/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830738.325792:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830738.325798:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830738.325803:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518316416:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830738.325810:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518316544:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830738.325835:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830738.325849:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830738/real 1787830738] req@ffff8b2e88042d00 x1874672518316544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830738.325862:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88042d00 x1874672518316544/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830738.325871:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830738.325874:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830738.325877:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88042d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518316544:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830738.325883:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518316672:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830738.325899:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830738.325907:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830738/real 1787830738] req@ffff8b2e88040780 x1874672518316672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830738.325914:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040780 x1874672518316672/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830738.325919:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830738.325921:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830738.325923:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518316672:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830738.325926:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518316800:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830738.325935:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830738.325940:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830738/real 1787830738] req@ffff8b2e88043840 x1874672518316800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830738.325947:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88043840 x1874672518316800/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830738.325951:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830738.325953:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830738.325954:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88043840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518316800:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830738.325957:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518316928:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830738.325965:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830738.325986:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830738/real 1787830738] req@ffff8b2e880412c0 x1874672518316928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830738.326001:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880412c0 x1874672518316928/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830738.326013:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830738.326017:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830738.326020:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880412c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518316928:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830738.326025:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518317056:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830738.326043:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830738.326052:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830738/real 1787830738] req@ffff8b2e88040b40 x1874672518317056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830738.326061:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88040b40 x1874672518317056/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830763 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830738.326068:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830738.326071:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830738.326074:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88040b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518317056:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830738.644918:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830739.158223:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830739.158643:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830739.348909:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830739.349030:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830739.670953:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830740.180765:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830740.182259:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830740.373260:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830740.373849:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830740.692780:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830741.204797:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830741.207401:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830741.396774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830741.396806:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830741.716879:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830742.228716:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830742.228976:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830742.420841:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830742.420916:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830742.740832:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830743.252768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830743.252847:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830743.444797:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830743.445030:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830743.445071:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830743.445078:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830743.445081:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830743.445090:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3749 00000100:00080000:1.0:1787830743.445100:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830743.445107:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830743.445151:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077840 x1874672518317184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830743.445177:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830743.445180:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830743.445183:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830743.445186:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3749 00000100:00080000:1.0:1787830743.445189:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830743.445193:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830743.445203:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:2.0:1787830743.445341:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518317184:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830743.445399:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830743.445446:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830743/real 1787830743] req@ffff8b2e88077840 x1874672518317184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830743.445462:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077840 x1874672518317184/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830743.445472:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830743.445476:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830743.445479:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518317184:192.168.203.101@tcp:250:kworker.0 00000100:00100000:1.0:1787830743.445484:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880770c0 x1874672518317312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830743.445493:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830743.445495:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830743.445497:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830743.445500:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3749 00000100:00100000:2.0:1787830743.445500:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518317312:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830743.445504:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000800:00000100:2.0:1787830743.445521:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830743.445534:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830743/real 1787830743] req@ffff8b2e880770c0 x1874672518317312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830743.445545:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880770c0 x1874672518317312/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830743.445553:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830743.445556:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830743.445559:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880770c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518317312:192.168.203.101@tcp:38:kworker.0 00000100:00080000:1.0:1787830743.445701:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830743.445707:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830743.445719:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880761c0 x1874672518317440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830743.445736:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830743.445739:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830743.445741:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830743.445744:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3749 00000100:00080000:1.0:1787830743.445748:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830743.445751:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830743.445754:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830743.445772:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074000 x1874672518317568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830743.445780:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830743.445783:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830743.445785:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830743.445788:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3749 00000100:00080000:1.0:1787830743.445790:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830743.445793:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830743.445797:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830743.445804:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074b40 x1874672518317696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830743.445812:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830743.445815:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830743.445816:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830743.445819:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3749 00000100:00080000:1.0:1787830743.445821:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830743.445824:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830743.445827:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830743.445834:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075e00 x1874672518317824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830743.445841:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830743.445844:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830743.445845:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830743.445849:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3749 00000100:00080000:1.0:1787830743.445963:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830743.445967:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830743.445972:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830743.445984:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077480 x1874672518317952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830743.446324:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518317440:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830743.446350:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830743.446365:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830743/real 1787830743] req@ffff8b2e880761c0 x1874672518317440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830743.446376:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880761c0 x1874672518317440/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830743.446384:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830743.446388:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830743.446391:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880761c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518317440:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830743.446396:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518317568:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830743.446415:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830743.446426:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830743/real 1787830743] req@ffff8b2e88074000 x1874672518317568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830743.446437:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074000 x1874672518317568/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830743.446444:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830743.446447:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830743.446451:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074000 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518317568:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830743.446456:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518317696:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830743.446475:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830743.446487:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830743/real 1787830743] req@ffff8b2e88074b40 x1874672518317696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830743.446497:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074b40 x1874672518317696/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830743.446504:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830743.446506:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830743.446509:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074b40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518317696:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830743.446514:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518317824:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830743.446529:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830743.446540:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830743/real 1787830743] req@ffff8b2e88075e00 x1874672518317824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830743.446549:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075e00 x1874672518317824/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830743.446562:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830743.446565:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830743.446568:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518317824:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830743.446573:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518317952:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830743.446591:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830743.446600:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830743/real 1787830743] req@ffff8b2e88077480 x1874672518317952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830743.446611:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077480 x1874672518317952/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830768 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830743.446619:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830743.446621:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830743.446625:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518317952:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830743.764862:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830744.276885:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:1.0:1787830744.276904:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830744.468796:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830744.468796:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830744.788834:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830745.300725:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830745.300842:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830745.492790:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830745.493257:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830745.812844:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:1.0:1787830746.324780:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830746.324995:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830746.516778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830746.516779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830746.836762:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830747.349004:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830747.349453:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830747.540760:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830747.540856:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830747.860744:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830748.372865:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830748.375227:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00080000:1.0:1787830748.564773:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830748.564778:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:1.0:1787830748.564787:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830748.564792:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830748.564801:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3754 00000100:00080000:1.0:1787830748.564812:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830748.564819:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830748.564858:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518318080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830748.564895:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830748.564908:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830748.564909:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830748.564914:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3754 00000100:00080000:1.0:1787830748.564918:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830748.564921:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830748.564931:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830748.564944:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518318208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830748.564952:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830748.564955:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830748.564957:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830748.564960:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3754 00000100:00080000:1.0:1787830748.564963:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830748.564966:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830748.564970:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830748.564979:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518318336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830748.564986:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830748.564989:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830748.564990:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830748.564993:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3754 00000100:00080000:1.0:1787830748.564995:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830748.564998:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830748.565002:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830748.565009:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518318464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830748.565016:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830748.565018:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830748.565020:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830748.565023:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3754 00000100:00080000:1.0:1787830748.565025:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830748.565028:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830748.565031:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830748.565038:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518318592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830748.565045:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830748.565048:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830748.565049:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830748.565052:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3754 00000100:00080000:1.0:1787830748.565054:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830748.565057:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830748.565061:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830748.565068:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518318720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830748.565075:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830748.565078:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830748.565079:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830748.565082:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3754 00000100:00080000:1.0:1787830748.565085:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830748.565088:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830748.565092:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830748.565099:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518318848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:3.0:1787830748.565671:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:2.0:1787830748.565959:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518318080:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830748.566023:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830748.566070:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830748/real 1787830748] req@ffff8b2e88075680 x1874672518318080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830748.566087:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518318080/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830748.566097:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830748.566101:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830748.566105:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518318080:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830748.566184:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518318208:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830748.566210:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830748.566222:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830748/real 1787830748] req@ffff8b2e88074780 x1874672518318208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830748.566234:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518318208/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830748.566242:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830748.566245:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830748.566249:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518318208:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830748.566256:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518318336:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830748.566274:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830748.566284:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830748/real 1787830748] req@ffff8b2e88074f00 x1874672518318336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830748.566296:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518318336/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830748.566303:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830748.566306:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830748.566309:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518318336:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830748.566315:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518318464:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830748.566332:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830748.566342:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830748/real 1787830748] req@ffff8b2e88075a40 x1874672518318464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830748.566353:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518318464/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830748.566361:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830748.566364:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830748.566367:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518318464:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830748.566373:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518318592:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830748.566389:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830748.566398:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830748/real 1787830748] req@ffff8b2e88076d00 x1874672518318592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830748.566409:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518318592/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830748.566417:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830748.566420:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830748.566423:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518318592:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830748.566429:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518318720:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830748.566446:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830748.566455:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830748/real 1787830748] req@ffff8b2e88077c00 x1874672518318720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830748.566472:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518318720/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830748.566479:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830748.566482:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830748.566485:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518318720:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830748.566491:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518318848:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830748.566507:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830748.566516:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830748/real 1787830748] req@ffff8b2e88076940 x1874672518318848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830748.566526:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518318848/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830773 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830748.566533:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830748.566536:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830748.566539:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518318848:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830748.885476:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830749.396782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830749.400781:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830749.588776:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830749.589327:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830749.908789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830750.420812:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830750.422240:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830750.612788:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830750.613002:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830750.932835:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830751.444762:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830751.445113:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830751.636775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830751.639268:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830751.956793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830752.468923:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830752.469124:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:3.0:1787830752.660777:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830752.661414:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830752.980818:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830753.493195:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830753.493774:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00080000:3.0:1787830753.687047:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830753.687059:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830753.687064:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830753.687073:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3760 00000100:00080000:3.0:1787830753.687088:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830753.687094:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830753.687140:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518318976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830753.687165:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830753.687168:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830753.687170:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830753.687174:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3760 00000100:00080000:3.0:1787830753.687177:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830753.687181:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830753.687190:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830753.687202:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518319104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830753.687211:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830753.687214:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830753.687228:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830753.687232:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3760 00000100:00080000:3.0:1787830753.687235:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830753.687238:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830753.687243:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830753.687252:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518319232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830753.687261:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830753.687264:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830753.687265:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830753.687269:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3760 00000100:00080000:3.0:1787830753.687272:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830753.687274:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830753.687278:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830753.687288:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518319360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830753.687296:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830753.687298:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830753.687300:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830753.687304:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3760 00000100:00080000:3.0:1787830753.687306:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830753.687309:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830753.687313:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830753.687321:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518319488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830753.687329:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830753.687332:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830753.687334:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830753.687337:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3760 00000100:00080000:3.0:1787830753.687339:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830753.687342:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830753.687345:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830753.687353:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518319616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830753.687366:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830753.687369:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830753.687370:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830753.687373:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3760 00000100:00080000:3.0:1787830753.687377:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830753.687379:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830753.687383:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830753.687392:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518319744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830753.687674:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518318976:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830753.687731:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830753.687771:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830753/real 1787830753] req@ffff8b2e8803b0c0 x1874672518318976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830753.687788:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518318976/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830753.687797:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830753.687802:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830753.687802:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830753.687808:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518318976:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830753.687816:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518319104:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830753.687837:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830753.687850:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830753/real 1787830753] req@ffff8b2e8803b840 x1874672518319104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830753.687879:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518319104/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830753.687887:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830753.687890:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830753.687893:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518319104:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830753.687899:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518319232:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830753.687918:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830753.687929:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830753/real 1787830753] req@ffff8b2e880383c0 x1874672518319232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830753.687940:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518319232/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830753.687947:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830753.687950:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830753.687953:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518319232:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830753.687959:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518319360:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830753.687975:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830753.687985:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830753/real 1787830753] req@ffff8b2e88038f00 x1874672518319360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830753.687995:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518319360/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830753.688007:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830753.688010:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830753.688013:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518319360:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830753.688019:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518319488:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830753.688035:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830753.688045:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830753/real 1787830753] req@ffff8b2e88039e00 x1874672518319488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830753.688055:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518319488/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830753.688062:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830753.688064:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830753.688067:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518319488:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830753.688073:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518319616:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830753.688090:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830753.688101:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830753/real 1787830753] req@ffff8b2e8803b480 x1874672518319616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830753.688111:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518319616/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830753.688118:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830753.688121:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830753.688125:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518319616:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830753.688130:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518319744:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830753.688148:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830753.688159:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830753/real 1787830753] req@ffff8b2e88039680 x1874672518319744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830753.688170:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518319744/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830778 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830753.688177:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830753.688180:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830753.688183:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518319744:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830753.688514:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830754.004814:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830754.516799:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830754.518688:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830754.708777:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830754.709172:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830755.028776:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830755.540777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830755.541796:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830755.732774:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830755.732774:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830756.052778:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830756.564866:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830756.568372:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830756.756787:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830756.757442:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830757.076790:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830757.588876:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830757.588955:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:2.0:1787830757.780779:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830757.780779:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830758.100789:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830758.612738:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830758.614053:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830758.804965:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:3.0:1787830758.804992:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830758.804998:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830758.805003:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830758.805012:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3765 00000100:00080000:3.0:1787830758.805023:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830758.805031:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830758.805071:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039a40 x1874672518319872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830758.805100:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830758.805104:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830758.805106:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830758.805110:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3765 00000100:00080000:3.0:1787830758.805113:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830758.805116:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830758.805125:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830758.805138:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a580 x1874672518320000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830758.805146:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830758.805150:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830758.805151:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830758.805154:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3765 00000100:00080000:3.0:1787830758.805157:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830758.805161:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830758.805165:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830758.805175:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803a1c0 x1874672518320128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830758.805188:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830758.805192:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830758.805193:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830758.805196:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3765 00000100:00080000:3.0:1787830758.805199:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830758.805203:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830758.805207:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830758.805217:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880392c0 x1874672518320256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830758.805226:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00100000:0.0:1787830758.805321:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830758.805343:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830758.805346:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00100000:2.0:1787830758.805353:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518319872:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830758.805407:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830758.805451:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830758/real 1787830758] req@ffff8b2e88039a40 x1874672518319872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830758.805920:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3765 00000100:00080000:3.0:1787830758.805925:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830758.805930:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830758.805937:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830758.805953:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518320384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830758.805964:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830758.805968:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830758.805970:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830758.805973:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3765 00000100:00080000:3.0:1787830758.805977:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830758.805981:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830758.805986:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830758.805996:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518320512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830758.806001:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039a40 x1874672518319872/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:3.0:1787830758.806012:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830758.806018:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:2.0:1787830758.806019:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:3.0:1787830758.806020:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:2.0:1787830758.806026:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00080000:3.0:1787830758.806029:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3765 00000100:00100000:2.0:1787830758.806034:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518319872:192.168.203.101@tcp:250:kworker.0 00000100:00080000:3.0:1787830758.806035:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830758.806043:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830758.806048:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830758.806058:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518320640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830758.806107:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518320000:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830758.806135:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830758.806149:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830758/real 1787830758] req@ffff8b2e8803a580 x1874672518320000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830758.806162:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a580 x1874672518320000/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830758.806172:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830758.806175:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830758.806179:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a580 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518320000:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830758.806185:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518320128:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830758.806208:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830758.806219:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830758/real 1787830758] req@ffff8b2e8803a1c0 x1874672518320128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830758.806231:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803a1c0 x1874672518320128/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830758.806239:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830758.806242:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830758.806246:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803a1c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518320128:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830758.806251:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518320256:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830758.806268:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830758.806277:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830758/real 1787830758] req@ffff8b2e880392c0 x1874672518320256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830758.806288:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880392c0 x1874672518320256/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830758.806397:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830758.806402:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830758.806406:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880392c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518320256:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830758.806432:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518320384:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830758.806453:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830758.806464:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830758/real 1787830758] req@ffff8b2e88039680 x1874672518320384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830758.806477:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518320384/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830758.806485:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830758.806488:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830758.806491:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518320384:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830758.806497:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518320512:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830758.806512:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830758.806522:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830758/real 1787830758] req@ffff8b2e8803b480 x1874672518320512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830758.806533:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518320512/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830758.806541:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830758.806544:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830758.806547:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518320512:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830758.806553:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518320640:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830758.806567:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830758.806576:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830758/real 1787830758] req@ffff8b2e88039e00 x1874672518320640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830758.806587:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518320640/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830783 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830758.806595:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830758.806598:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830758.806601:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518320640:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830759.124809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830759.636777:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830759.637000:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830759.828854:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830759.830679:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830760.148860:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830760.660898:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830760.661002:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830760.852795:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830760.852833:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830761.172809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830761.684794:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830761.684855:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830761.876782:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830761.876916:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830762.197417:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830762.708784:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830762.708964:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830762.901124:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830762.902034:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830763.220875:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830763.732930:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830763.733204:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830763.924775:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830763.925630:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00080000:1.0:1787830763.925866:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830763.925872:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:1.0:1787830763.925876:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830763.925886:0:38:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3770 00000100:00080000:1.0:1787830763.925897:0:38:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830763.925905:0:38:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:1.0:1787830763.925946:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076940 x1874672518320768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830763.925978:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830763.925982:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830763.925984:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830763.925988:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3770 00000100:00080000:1.0:1787830763.925991:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830763.925995:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830763.926005:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830763.926016:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88077c00 x1874672518320896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830763.926025:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830763.926029:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830763.926031:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830763.926034:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3770 00000100:00080000:1.0:1787830763.926038:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830763.926041:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830763.926046:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:1.0:1787830763.926056:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88076d00 x1874672518321024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830763.926065:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830763.926068:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830763.926069:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830763.926073:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3770 00000100:00080000:1.0:1787830763.926075:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830763.926078:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830763.926082:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830763.926091:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075a40 x1874672518321152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830763.926100:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830763.926102:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:1.0:1787830763.926104:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830763.926107:0:38:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3770 00000100:00080000:1.0:1787830763.926110:0:38:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830763.926114:0:38:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830763.926119:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830763.926128:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074f00 x1874672518321280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830763.926137:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830763.926141:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:1.0:1787830763.926143:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830763.926146:0:38:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3770 00000100:00080000:1.0:1787830763.926150:0:38:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830763.926153:0:38:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830763.926157:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830763.926167:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88074780 x1874672518321408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:1.0:1787830763.926176:0:38:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:1.0:1787830763.926179:0:38:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:1.0:1787830763.926181:0:38:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:1.0:1787830763.926184:0:38:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3770 00000100:00080000:1.0:1787830763.926187:0:38:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:1.0:1787830763.926191:0:38:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:1.0:1787830763.926195:0:38:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:1.0:1787830763.926204:0:38:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88075680 x1874672518321536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:2.0:1787830763.926372:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518320768:192.168.203.101@tcp:250:kworker.0 00000800:00000100:2.0:1787830763.926429:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830763.926473:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830763/real 1787830763] req@ffff8b2e88076940 x1874672518320768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830763.926487:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076940 x1874672518320768/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830763.926497:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830763.926502:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830763.926505:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076940 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518320768:192.168.203.101@tcp:250:kworker.0 00000100:00100000:2.0:1787830763.926512:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518320896:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830763.926534:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830763.926546:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830763/real 1787830763] req@ffff8b2e88077c00 x1874672518320896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830763.926574:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88077c00 x1874672518320896/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830763.926582:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830763.926586:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830763.926588:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88077c00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518320896:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830763.926594:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518321024:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830763.926614:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830763.926626:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830763/real 1787830763] req@ffff8b2e88076d00 x1874672518321024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830763.926636:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88076d00 x1874672518321024/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830763.926643:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830763.926674:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830763.926677:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88076d00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518321024:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830763.926683:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518321152:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830763.926815:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830763.926828:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830763/real 1787830763] req@ffff8b2e88075a40 x1874672518321152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830763.926840:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075a40 x1874672518321152/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830763.926853:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830763.926856:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830763.926859:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075a40 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518321152:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830763.926864:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518321280:192.168.203.101@tcp:38:kworker.0 00000800:00000100:2.0:1787830763.926884:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830763.926895:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830763/real 1787830763] req@ffff8b2e88074f00 x1874672518321280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830763.926905:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074f00 x1874672518321280/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830763.926913:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830763.926916:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830763.926919:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518321280:192.168.203.101@tcp:38:kworker.0 00000100:00100000:2.0:1787830763.926925:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518321408:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830763.926944:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830763.926956:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830763/real 1787830763] req@ffff8b2e88074780 x1874672518321408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830763.926967:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88074780 x1874672518321408/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830763.926974:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830763.926976:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830763.926980:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88074780 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518321408:192.168.203.101@tcp:8:kworker.0 00000100:00100000:2.0:1787830763.926985:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518321536:192.168.203.101@tcp:8:kworker.0 00000800:00000100:2.0:1787830763.927003:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:2.0:1787830763.927014:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830763/real 1787830763] req@ffff8b2e88075680 x1874672518321536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830763.927025:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88075680 x1874672518321536/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830788 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:2.0:1787830763.927032:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:2.0:1787830763.927034:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:2.0:1787830763.927037:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88075680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518321536:192.168.203.101@tcp:8:kworker.0 00000100:00100000:3.0:1787830764.244799:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830764.756768:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830764.758821:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830764.948793:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830764.948895:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:3.0:1787830765.269812:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830765.780792:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830765.781165:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830765.972789:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830765.973073:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830766.293220:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830766.805167:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830766.805335:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:0.0:1787830766.996789:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:3.0:1787830766.997315:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830767.318206:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830767.828782:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830767.831096:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:3.0:1787830768.020801:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830768.021370:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830768.341568:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:1.0:1787830768.852839:0:14479:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830768.852901:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830769.044980:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00080000:3.0:1787830769.045848:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) b58dbfc8-99f9-40e7-8102-506d8349451a->MGS: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830769.045855:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) MGS: starting recovery 00000100:00080000:3.0:1787830769.045859:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830769.045868:0:36:0:(import.c:510:import_select_connection()) MGC192.168.203.101@tcp: connect to NID 192.168.203.101@tcp last attempt 3775 00000100:00080000:3.0:1787830769.045878:0:36:0:(import.c:589:import_select_connection()) MGC192.168.203.101@tcp: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830769.045885:0:36:0:(import.c:624:import_select_connection()) MGC192.168.203.101@tcp: import ffff8b2e88a41800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000100:00100000:3.0:1787830769.045921:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88038f00 x1874672518321664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830769.045951:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830769.045954:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830769.045956:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830769.045960:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3775 00000100:00080000:3.0:1787830769.045963:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830769.045967:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e901ad800: import ffff8b2e84b10800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830769.045976:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830769.045990:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e880383c0 x1874672518321792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830769.045999:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830769.046002:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830769.046004:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830769.046007:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3775 00000100:00080000:3.0:1787830769.046009:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830769.046013:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e901ad800: import ffff8b2e88a45800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830769.046017:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 1703936, lost: 0. 00000100:00100000:3.0:1787830769.046028:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b840 x1874672518321920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830769.046037:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 31a8f82c-9690-4198-830f-493f54020135->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830769.046039:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830769.046041:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830769.046044:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: connect to NID 192.168.203.101@tcp last attempt 3775 00000100:00080000:3.0:1787830769.046047:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830769.046051:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e901ad800: import ffff8b2e88a47800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830769.046055:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830769.046065:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b0c0 x1874672518322048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830769.046072:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-MDT0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830769.046075:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-MDT0000_UUID: starting recovery 00000100:00080000:3.0:1787830769.046077:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830769.046080:0:36:0:(import.c:510:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3775 00000100:00080000:3.0:1787830769.046083:0:36:0:(import.c:589:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830769.046087:0:36:0:(import.c:624:import_select_connection()) lustre-MDT0000-mdc-ffff8b2e84abe800: import ffff8b2e87378000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830769.046091:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xae7e7e7be374d3b8 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830769.046101:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039e00 x1874672518322176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830769.046109:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0000_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830769.046112:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0000_UUID: starting recovery 00000100:00080000:3.0:1787830769.046113:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830769.046117:0:36:0:(import.c:510:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3775 00000100:00080000:3.0:1787830769.046120:0:36:0:(import.c:589:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830769.046123:0:36:0:(import.c:624:import_select_connection()) lustre-OST0000-osc-ffff8b2e84abe800: import ffff8b2e84b13800 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830769.046127:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830769.046136:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e8803b480 x1874672518322304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00080000:3.0:1787830769.046144:0:36:0:(pinger.c:229:ptlrpc_pinger_process_import()) 8e7ce795-6dad-45e6-b76e-c5bf3324e96e->lustre-OST0001_UUID: level DISCONN/3 force 0 force_next 0 deactive 0 pingable 1 suppress 0 00000100:00080000:3.0:1787830769.046147:0:36:0:(pinger.c:237:ptlrpc_pinger_process_import()) lustre-OST0001_UUID: starting recovery 00000100:00080000:3.0:1787830769.046148:0:36:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from DISCONN to CONNECTING 00000100:00080000:3.0:1787830769.046151:0:36:0:(import.c:510:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: connect to NID 192.168.203.101@tcp last attempt 3775 00000100:00080000:3.0:1787830769.046154:0:36:0:(import.c:589:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: tried all connections, increasing latency to 20s 00000100:00080000:3.0:1787830769.046157:0:36:0:(import.c:624:import_select_connection()) lustre-OST0001-osc-ffff8b2e84abe800: import ffff8b2e90dee000 using connection 192.168.203.101@tcp/192.168.203.101@tcp 00000008:00100000:3.0:1787830769.046161:0:36:0:(osc_request.c:4019:osc_reconnect()) ocd_connect_flags: 0xa0467af2e3640038 ocd_version: 34683201 ocd_grant: 2097152, lost: 0. 00000100:00100000:3.0:1787830769.046171:0:36:0:(import.c:826:ptlrpc_connect_import_locked()) @@@ (re)connect request (timeout 5) req@ffff8b2e88039680 x1874672518322432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 0 dl 0 ref 1 fl New:NQU/0/ffffffff rc 0/-1 job:'' uid:4294967295 gid:4294967295 projid:4294967295 00000100:00100000:0.0:1787830769.046193:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518321664:192.168.203.101@tcp:250:kworker.0 00000800:00000100:0.0:1787830769.046264:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830769.046306:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830769/real 1787830769] req@ffff8b2e88038f00 x1874672518321664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830769.046331:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88038f00 x1874672518321664/t0(0) o250->MGC192.168.203.101@tcp@192.168.203.101@tcp:26/25 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830769.046340:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a41800 MGS: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830769.046344:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of MGS on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830769.046348:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88038f00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:b58dbfc8-99f9-40e7-8102-506d8349451a:2346:1874672518321664:192.168.203.101@tcp:250:kworker.0 00000100:00100000:0.0:1787830769.046355:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518321792:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830769.046378:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830769.046390:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830769/real 1787830769] req@ffff8b2e880383c0 x1874672518321792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830769.046403:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e880383c0 x1874672518321792/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e901ad800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830769.046412:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b10800 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830769.046415:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830769.046418:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e880383c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518321792:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830769.046424:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518321920:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830769.046443:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830769.046454:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830769/real 1787830769] req@ffff8b2e8803b840 x1874672518321920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830769.046465:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b840 x1874672518321920/t0(0) o8->lustre-OST0000-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830769.046477:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a45800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830769.046480:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830769.046483:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b840 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518321920:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830769.046489:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518322048:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830769.046508:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830769.046518:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830769/real 1787830769] req@ffff8b2e8803b0c0 x1874672518322048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830769.046529:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b0c0 x1874672518322048/t0(0) o8->lustre-OST0001-osc-ffff8b2e901ad800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830769.046536:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e88a47800 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830769.046539:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830769.046542:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b0c0 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:31a8f82c-9690-4198-830f-493f54020135:2346:1874672518322048:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830769.046548:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518322176:192.168.203.101@tcp:38:kworker.0 00000800:00000100:0.0:1787830769.046573:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830769.046585:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830769/real 1787830769] req@ffff8b2e88039e00 x1874672518322176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:2.0:1787830769.046594:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830769.046598:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039e00 x1874672518322176/t0(0) o38->lustre-MDT0000-mdc-ffff8b2e84abe800@192.168.203.101@tcp:12/10 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830769.046609:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e87378000 lustre-MDT0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830769.046613:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-MDT0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830769.046616:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039e00 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518322176:192.168.203.101@tcp:38:kworker.0 00000100:00100000:0.0:1787830769.046622:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518322304:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830769.046723:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830769.046735:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830769/real 1787830769] req@ffff8b2e8803b480 x1874672518322304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830769.046746:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e8803b480 x1874672518322304/t0(0) o8->lustre-OST0000-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830769.046754:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e84b13800 lustre-OST0000_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830769.046768:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0000_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830769.046771:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e8803b480 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518322304:192.168.203.101@tcp:8:kworker.0 00000100:00100000:0.0:1787830769.046777:0:2346:0:(client.c:1919:ptlrpc_send_new_req()) Sending RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518322432:192.168.203.101@tcp:8:kworker.0 00000800:00000100:0.0:1787830769.046798:0:2346:0:(socklnd_cb.c:906:ksocknal_launch_packet()) No usable routes to 12345-192.168.203.101@tcp 00000100:00100000:0.0:1787830769.046809:0:2346:0:(client.c:2503:ptlrpc_expire_one_request()) @@@ Request sent has failed due to network error: [sent 1787830769/real 1787830769] req@ffff8b2e88039680 x1874672518322432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQr/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00100000:0.0:1787830769.046821:0:2346:0:(client.c:2534:ptlrpc_expire_one_request()) @@@ err -110, sent_state=CONNECTING (now=CONNECTING) req@ffff8b2e88039680 x1874672518322432/t0(0) o8->lustre-OST0001-osc-ffff8b2e84abe800@192.168.203.101@tcp:28/4 lens 520/544 e 0 to 1 dl 1787830794 ref 1 fl Rpc:eXNQU/200/ffffffff rc 0/-1 job:'kworker.0' uid:0 gid:0 projid:4294967295 00000100:00080000:0.0:1787830769.046828:0:2346:0:(import.c:65:import_set_state_nolock()) ffff8b2e90dee000 lustre-OST0001_UUID: changing import state from CONNECTING to DISCONN 00000100:00080000:0.0:1787830769.046832:0:2346:0:(import.c:1511:ptlrpc_connect_interpret()) recovery of lustre-OST0001_UUID on 192.168.203.101@tcp failed (-110) 00000100:00100000:0.0:1787830769.046834:0:2346:0:(client.c:2414:ptlrpc_check_set()) Completed RPC req@ffff8b2e88039680 pname:cluuid:pid:xid:nid:opc:job ptlrpcd_rcv:8e7ce795-6dad-45e6-b76e-c5bf3324e96e:2346:1874672518322432:192.168.203.101@tcp:8:kworker.0 00000100:00100000:1.0:1787830769.364793:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00000100:00100000:2.0:1787830769.876925:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00010000:00010000:3.0:1787830769.878733:0:5577:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830770.068878:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:0.0:1787830770.069009:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:1.0:1787830770.388964:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds 00010000:00010000:3.0:1787830770.900910:0:6036:0:(ldlm_lockd.c:2804:ldlm_bl_get_work()) No blwi found in queue (no bl locks in queue) 00000100:00100000:2.0:1787830770.900916:0:14480:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e88db7e40 going to sleep for 0 seconds 00000100:00100000:0.0:1787830771.092820:0:21402:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2000 going to sleep for 0 seconds 00000100:00100000:2.0:1787830771.093917:0:21400:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e873b2840 going to sleep for 0 seconds 00000100:00100000:1.0:1787830771.412809:0:21412:0:(client.c:2715:ptlrpc_set_wait()) set ffff8b2e90eed900 going to sleep for 0 seconds Debug log: 80063 lines, 80063 kept, 0 dropped, 0 bad.